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EXECUTIVE  SUMMARY 


The  Terminal  Convective  Weather  Forecast  (TCWF)  2000  Demonstration  was  conducted 
during  the  convective  summer  season  (between  June  and  October  2000)  to  evaluate  the 
product’s  operational  suitability  for  Air  Traffic  Control  (ATC)  operations.  The 
demonstration  was  planned  and  conducted  by  test  personnel  from  the  FAA  William  J. 
Hughes  Technical  Center  (hereafter  referred  to  as  Technical  Center),  Weather  Group 
(ACB-630)  and  funded  by  the  FAA  Aviation  Weather  Research  Program  (AWRP).  The 
Massachusetts  Institute  of  Technology  Lincoln  Laboratory  (MIT/LL)  developed  the 
TCWF,  which  has  undergone  two  previous  iterations  of  operational  and  usability  testing. 

Demonstration  sites  included  the  Memphis  International  Airport  (MEM)  Terminal  Radar 
Approach  Control  (TRACON)  and  the  Memphis  Air  Route  Traffic  Control  Center 
(ARTCC)  (ZME)  Traffic  Management  Unit  (TMU).  Participants  included  MEM 
TRACON  Supervisors,  and  ZME  TMU  Traffic  Management  Coordinators  (TMCs). 

The  purpose  of  the  demonstration  was  to  ascertain  the  TCWF:  overall  utility;  ease  of 
use;  readability;  and  perceived  benefit  to  ATC  tasking  as  it  relates  to  convective  weather 
events. 

The  demonstration  was  conducted  in  two  phases  in  order  to  capture  both  air-mass  and 
line-storm  convection.  In  this  way,  user  impressions  of  the  product  during  both  of  these 
weather  phenomena  could  be  captured.  Data  collection  metrics  included:  questionnaires, 
structured  interviews,  and  workload  scales. 

Results  from  both  data  collection  phases  were  similar.  Overall  impressions  of  the  TCWF 
were  positive  for  both  user  groups.  Differences  in  perceptions  from  data  collected  during 
Phase  1  and  2  were  not  significantly  different. 

All  MEM  TRACON  users  reported  the  TCWF  provided  benefit  in  performing  tasks 
during  operationally  significant  convective  events,  although  Phase  2  results  indicated  a 
slightly  less  positive  view  (extremely  useful  vs.  somewhat  useful).  In  either  phase,  none 
of  the  task  areas  were  rated  negatively  (i.e.  no  use).  Also,  for  some  task  areas,  frequency 
of  use  declined  somewhat  from  Phase  1  to  Phase  2  (most  of  the  time  vs.  now  and  then). 

ZME  TMCs’  perceptions  for  tasks  were  mostly  positive.  Although  found  very  useful  for 
MEM  airspace  impacts,  the  TCWF  was  not  adequate  for  all  airspace  requirements,  which 
comprises  a  majority  of  ZME  operations. 

Almost  all  users  viewed  the  TCWF  as  a  beneficial  supplement  to  the  Integrated  Terminal 
Weather  System  (ITWS),  where  users  reportedly  preferred  the  TCWF  areal  weather 
coverage  and  60-minute  animated  forecast  over  the  ITWS  Storm  Extrapolated  Position 
(SEP)  20-minute  leading  edge  line  depictions. 

Overall  data  revealed  that  for  most  users  (MEM  and  ZME),  the  TCWF  enhanced 
situational  awareness,  and  for  MEM  TRACON  users,  may  have  contributed  to  a  decrease 


in  perceived  operator  mental  workload  in  the  performance  of  tasks  affected  by  convective 
weather.  Suggested  product  modifications  included  but  were  not  limited  to: 
improvement  of  forecast  accuracy,  especially  for  air-mass  convection;  both  separation 
from  and  integration  with  the  ITWS  display  platform;  and  use  of  a  different  color  scheme 
for  forecast  depictions. 
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1.  INTRODUCTION. 


1.1  BACKGROUND. 

The  Federal  Aviation  Administration  (FAA)  Aviation  Weather  Research  Program  (AWRP)  has 
sponsored  the  Massachusetts  Institute  of  Technology  Lincoln  Laboratories  (MIT/LL)  in  the 
development  of  a  Terminal  Convective  Weather  Forecast  (TCWF).  The  TCWF  graphically 
depicts  a  0  to  60-minute  forecast  of  convective  weather  for  the  terminal  area.  The  TCWF  is  in 
its  third  iteration  of  operational  demonstrations  resulting  in  the  current  Build  3. 

The  purpose  of  the  TCWF  2000  Demonstration  was  to  collect  user  feedback  regarding  the  utility, 
usability,  and  task  benefit  of  Build  3  in  order  to  effect  improvements  to  the  TCWF  forecast  and 
display.  Another  goal  was  to  provide  demonstration  results  to  the  FAA  Integrated  Terminal 
Weather  System  (ITWS)  Program  Office  for  consideration  of  the  TWCF  as  a  Pre-Planned 
Product  Improvement  (P3I)  for  testing  and  integration. 

During  the  summer  of  1998,  the  TCWF  was  demonstrated  at  the  Dallas/Ft.  Worth  International 
Airport  (DFW)  and  Ft.  Worth  Air  Route  Traffic  Control  Center  (ARTCC).  The  FAA  William  J. 
Hughes  Technical  Center  (hereafter  referred  to  as  Technical  Center)  and  MIT/LL  were 
responsible  for  the  conduct  of  this  demonstration.  Overall,  the  forecast  was  viewed  favorably  by 
Terminal  Radar  Approach  Control  (TRACON)  and  ARTCC  Traffic  Management  Coordinators 
(TMCs)  as  well  as  Center  Weather  Service  Unit  (CWSU)  Meteorologists.  The  forecast  was 
deemed  useful  for  a  number  of  task  areas.  Some  minor  display  modifications  were  requested 
along  with  a  request  by  ARTCC  TMCs  for  longer  forecast  times. 

During  the  1999  convective  weather  season  (June  through  October)  Build  2  was  again 
demonstrated  at  the  Dallas  TRACON  and  Dallas/Ft.  Worth  ARTCC.  Demonstrations  were  also 
conducted,  simultaneously,  at  the  Orlando  International  Airport  (MCO)  TRACON  and 
Jacksonville  ARTCC  (ZJX).  Users  consisted  of  TMCs,  Supervisors,  and  CWSUs.  Final  results 
from  the  1999  Demonstrations  indicated  product  perceptions  differed  by  location.  These 
perceptions  may  be  a  function  of  the  different  weather  phenomena  encountered  at  each  location. 

Based  on  results  from  the  1999  Demonstration  and  internal  product  monitoring  conducted  by 
MIT/LL,  changes  were  made  to  the  forecast  and  accuracy  scoring  algorithms.  For  the  year  2000 
convective  season,  the  TCWF  was  demonstrated  at  Memphis  International  Airport  (MEM) 
TRACON  and  Memphis  ARTCC  (ZME)  in  each  facility’s  respective  Traffic  Management  Unit 
(TMU).  Memphis  sites  were  chosen  because  users  were  already  familiar  with  using  ITWS 
Situation  Displays  (SDs)  but  had  not  been  exposed  to  the  TCWF  product.  Participants  included 
TMCs  and  Supervisors.  Also,  for  the  2000  Demonstration,  the  TCWF  was  displayed  on  an 
ITWS  SD  platform.  Although  not  integrated  with  ITWS,  the  display  looked  like  the  ITWS  SD 
display,  with  activated  alarms,  status  bars  and  Unix  type  graphical  user  interfaces  (GUIs).  This 
significantly  changed  the  product’s  appearance  from  the  1998  and  1999  Demonstrations. 

The  TCWF  2000  Demonstration,  conducted  in  two  phases,  endeavored  to  capture  data  on 
product  perceptions  during  different  types  of  convective  weather  activity  (air-mass  vs.  line-storm 
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convection).  Phase  1  data  collection  took  place  from  July  27  through  July  29,  2000;  Phase  2 
from  November  7  through  November  9,  2000. 

Throughout  the  remainder  of  this  report,  Memphis  International  Airport  and  Memphis  ARTCC 
will  be  referred  to  by  their  three-letter  designators;  MEM  and  ZME,  respectively. 

1.2  PURPOSE. 

The  purpose  of  this  report  is  to  summarize  and  report  results  on  data  collected  during  Phase  1 
and  Phase  2  at  MEM  TRACON  and  ZME.  Conclusions  and  recommendations  derived  from 
these  results  are  also  reported  with  a  description  of  the  TCWF  and  demonstration  overview. 

1.3  SCOPE. 

This  report  addresses  the  Phase  1  and  Phase  2  user  evaluation  portion  of  the  2000  TCWF 
Demonstration.  Additional  analysis  was  conducted  by  personnel  from  MIT/LL,  and  may  be 
submitted  as  a  separate  report. 

2.  REFERENCE  DOCUMENTS. 

FAA  Acquisition  Management  System,  Test  &  Evaluation  Process  Guidelines,  June  1999. 

Terminal  Convective  Weather  Forecast  Demonstration  Plan  and  Procedures,  May  2000,  FAA 
William  J.  Hughes  Technical  Center,  ACB-630. 

1999  Terminal  Convective  Weather  Forecast  Demonstration  Phase  1  &  2  Summary  Report, 
December  1999,  FAA  William  J.  Hughes  Technical  Center,  ACB-630. 

Demonstration  on  the  Usability  of  the  1999  Terminal  Convective  Weather  Forecast  (TCWF) 
Product  for  Air  Traffic  Control  Managers,  September  2000,  FAA  William  J.  Hughes  Technical 
Center,  ACB-630. 

1998  Terminal  Convective  Weather  Forecast  Demonstration  Summary  Report,  September  1998, 
FAA  William  J.  Hughes  Technical  Center,  ACB-630. 

Handbook  of  Human  Factors  Testing  and  Evaluation,  OBrien,  T.,  and  Charlton,  S.  (eds.)  1996. 

3.  SYSTEM/PRODUCT  OVERVIEW. 

3.1  SYSTEM  DESCRIPTION. 

The  TCWF  used  a  method,  labeled  as  the  Growth  and  Decay  Tracker,  to  determine  the  forecast 
of  convective  weather.  The  tracker  is  designed  to  separate  large-scale  (envelope)  motion  from 
small-scale  cell  motion  of  a  storm.  By  effectively  tracking  the  storm  scale  forcing,  the  forecast 
product  is  able  to  account  for  the  systematic  growth  and  decay  of  organized  convective  systems. 
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The  TCWF  algorithm  process  creates  forecasts,  out  to  60  minutes  incrementally,  in  ten-minute 
steps. 

TCWF  resolution  is  1  kilometer  (km)  for  ranges  between  5  and  50  nautical  miles  (nm).  A 
resolution  of  2  km  is  used  for  ranges  above  50  nm  (100  and  200  nm).  New  images  are  created 
approximately  every  6  minutes,  depending  on  when  the  Next  Generation  Radar  (NEXRAD) 
updates. 

Changes  to  the  TCWF  from  previous  versions  included  updates  to  the  accuracy-scoring 
algorithm.  The  criteria  for  this  "User  Scoring"  technique  included  a  forecast  that  was  accurate  to 
within  5  nm  of  actual  weather.  Overlaying  a  5x5  pixel  grid,  any  level  3  or  above  pixels  were 
scored  as  hits  and  level  2  pixels  as  partial  hits.  The  1999  algorithm  only  scored  the  level  3  and 
above  pixels.  Therefore,  accuracy  scores  should  appear  higher  for  Build  3.  A  thirty-minute 
running  average  of  these  scores  constituted  the  forecast  accuracy.  Separate  scoring  was  available 
for  both  the  TRACON  (50  nm)  and  longer  range  (200  nm)  views. 

For  Memphis  users,  the  TCWF  display  was  a  dedicated  display  and  was  delivered  via  leased 
phone  lines  from  the  Forecast  Product  Generator  located  at  MIT/LL. 

3.2  PRODUCT  OVERVIEW. 


3.2.1  Display  Window. 

The  display  window  graphically  presents  past,  current  and  forecast  convective  weather.  Display 
ranges  are  initially  displayed  in  two  separate  windows:  1)  the  TRACON  Range  (out  to  50  nm) 
and  2)  the  Long  Range  (out  to  200  nm).  Other  ranges  may  be  user  selected  and  are  discussed  in 
section  3.2.5.6  of  this  report.  Other  display  depictions,  i.e.  overlays,  windowing,  and  looping, 
are  user  selected  from  the  window  configuration  and  display  configuration  buttons.  These  items 
are  discussed  in  subsequent  sections  of  this  report.  Figure  1  shows  an  example  of  a  30-minute 
forecast  display  of  the  Memphis  TRACON  view. 

3.2.2  ITWS  Alerts. 


ITWS  alert  buttons  on  the  TCWF  display  are  identical  to  the  alerts  on  the  ITWS  display.  If  any 
alert  is  activated,  it  is  visible  on  both  displays.  The  TCWF  product  itself  does  not  affect  the 
ITWS  alerts.  ITWS  alert  buttons  are  shown  in  figure  1. 

3.2.3  Product  Status  Buttons. 


Product  status  buttons  are  located  on  the  top  left-hand  side  of  the  display,  directly  underneath 
display  configuration  buttons.  The  activated  buttons  show  the  status  of  the  TCWF  forecast 
product  as  well  as  the  ITWS  products  connected  to  the  alerts.  These  alerts  included: 

a.  Tornado  warning, 

b.  Lightning, 
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c.  Gust  front, 

d.  ASR  (Airport  Surveillance  Radar)  status, 

e.  ATIS  (Automatic  Information  Server)  status,  and  Forecast  status. 

The  status  button  will  be  red  if  the  product  is  not  available  for  the  airport  in  the  active  graphics 
window.  If  the  product  is  available,  the  button  will  be  green.  Figure  1  depicts  product  status 
buttons. 

3.2.4  Display  Configuration  Buttons. 

Display  configuration  buttons  are  located  on  the  top  left  hand  side  of  the  TCWF  display  (see 
figure  1)  and  are  identical  to  the  ITWS  display  configuration  buttons.  However,  only  the 
Window'  option  is  available  on  the  TCWF  display  since  the  ITWS  graphics  window  has  been 
disabled. 

3.2.4, 1  Window  Option. 

Choosing  the  window  option  will  allow  for  the  selection  of  new  forecast  windows.  Up  to  four 
forecast  windows  can  be  displayed  for  the  Memphis  airport.  Users  have  the  option  to  open  a 
new  window,  restore  a  previous  window,  or  close  the  current  window. 


3.2.5  Window  Configuration  Buttons. 

Window  configuration  buttons  are  located  on  the  right  hand  side  of  the  display  window.  Each 
letter  or  symbol  on  the  menu  bar  represents  a  display  function  using  standard  Unix  symbology. 
For  example,  Z  means  "Zoom  In"  and  "O"  means  "Overlay  Menu".  The  window  configuration 
buttons  are  identical  to  those  on  the  ITWS  display  with  the  following  exceptions: 

a.  Color-bar  Toggle 

b.  Forecast  Accuracy 

c.  Loop  Settings,  and 

d.  Start/Stop  Loop  Control. 
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Display  Configuration  Buttons 


FIGURE  1.  TCWF  MEMPHIS  DISPLAY  WINDOW 


Each  window  configuration  button  is  described  below  and  depicted  in  figure  2.  Options  that  are 
italicized  are  unique  to  the  TCWF  display. 

Product  Interaction  Mode  (arrow) 

Pan  Interaction  Mode  (hand) 

Zoom  In  (Z) 


Zoom  Out  (Un-zoom)  (U) 

Home  (house) 

Re-center  Interaction  Mode  (target) 
Range  Menu  (R) 

Overlay  Menu  (O) 

Colorbar  Toggle  (C) 

Forecast  Accuracy  (%) 

Loop  Settings  (L) 

Start/Stop  Loop  Control  (circular  arrow) 


FIGURE  2.  WINDOW  CONFIGURATION  BUTTONS 


3.2.5. 1  Product  Interaction  Mode. 


Product  interaction  mode,  represented  by  an  arrow  icon,  allows  cursor  movement  within  a 
specified  window. 

3.2.5.2  Pan  Interaction  Mode. 

Pan  interaction  mode,  represented  by  a  hand  icon,  enables  movement  of  graphics  within  a 
specified  display  window. 

3.2.5.3  Zoom  In  and  Zoom  Out. 


Zoom  in,  or  "Z"  and  Zoom  Out,  or  "U"  for  "unzoom"  incrementally  zooms  in  or  out  of  a 
graphical  display  window. 

3.2.5.4  Home. 


The  home  option,  or  house  icon,  will  return  a  display  graphic  to  its  original  default  presentation. 
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3 . 2 . 5 . 5  Re-center  Interaction  Mode. 


The  re-center  interaction  mode  option,  represented  by  a  target  symbol,  will  automatically  center 
a  graphic  within  a  display  window. 

3.2.5.6  Range  Menu. 

The  range  selection  buttons  allow  the  user  to  select  the  range  of  the  display.  Ranges  between 
5  nm  and  TRACON  (50  nm)  will  use  the  1  km  product.  The  100  nm  and  200  nm  ranges  will  use 
the  2  km  product.  Figure  3  depicts  the  range  menu  display. 


FIGURE  3.  RANGE  SELECTION  MENU 


3.2.5.7  Overlay  Menu. 

Geographical  and  areal  locators  may  be  overlaid  on  the  active  display  window.  By  clicking  on 
the  overlay  option  the  user  may  select  or  deselect  any  of  the  following: 

a.  Range  rings 

b.  Navigational  fixes 

c.  Outlying  airports 

d.  Roads 

e.  Counties 

f.  Outer  mark 

g.  Victor  ways 

h.  Jet  ways 

i.  High  altitude  select 

j.  Low  altitude  select 
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3.2.5.8  Color-bar  Toggle. 


The  color-bar  toggle  scale  (levels  1-6)  indicates  past  weather.  This  scale  is  identical  to  the 
ITWS  6  level  weather  intensity  color  scale. 


The  predicted  or  forecast  weather  key  shows  regions  in  which  the  TCWF  expects  a  moderate  and 
high  probability  that  level  3  or  above  weather  (yellow,  orange  and  red  on  the  ITWS  scale)  will 
occur  at  the  forecast  time  interval.  Solid  yellow  is  used  for  high  probability  regions  and  a 
hatched  yellow/gray  is  used  for  moderate  probability  regions.  Figure  4  graphically  portrays  the 
color-bar  toggle. 


MOD  HHIH 


FIGURE  4.  COLOR-BAR  TOGGLE 


3.2.5.9  Forecast  Accuracy. 

The  forecast  accuracy  or  scoring  display  is  illustrated  in  figure  5.  This  display  provides  users 
with  real-time  scoring  information  of  TCWF  accuracy.  After  forecasts  have  existed  for  30  and 
60  minutes,  the  forecast  image  is  compared  to  the  “truth”  image  or  actual  radar  image  and  the 
accuracy  of  the  forecast  is  scored.  Past  performance  does  not  indicate  the  accuracy  of  the  current 
forecast.  Accuracy  scores  are  available  for  both  TRACON  and  200  nm  ranges. 


FIGURE  5.  FORECAST  ACCURACY  DISPLAY 
3.2.5.10  Loop  Settings. 

The  movie  loop  selections,  illustrated  in  figure  6,  allow  the  user  to  select  a  variety  of  movie 
loops  for  displaying  data.  Loops  range  from  past  weather  only  to  30  or  60-minute  forecasts. 
Past  weather  (back  to  30  minutes)  and  forecasted  weather  can  be  looped  together  and  are  shown 
in  10-minute  increments.  Looping  can  occur  with  or  without  past  weather.  All  images  update 
automatically  when  new  forecasts  become  available. 
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Three  speed  selections  (slow,  moderate,  fast)  are  also  available  so  users  may  adjust  the  speed  of 
the  animation. 


FIGURE  6.  LOOP  SETTINGS 
3.2.5. 1 1  Start/Stop  Loop  Control. 

The  start/stop  loop  control  function,  shown  in  figure  7,  controls  the  starting  and  stopping  of 
forecast  loops.  It  also  enables  the  user  to  display  static  forecast  images. 


FIGURE  7.  START/STOP  LOOP  CONTROL 
3.2.6  Relative  Forecast  Time. 

Figure  8  depicts  the  relative  forecast  time  on  the  TCWF  display.  This  is  "relative  time",  or  the 
time  difference  from  the  current  time  to  the  time  of  the  weather  shown  in  the  display  window. 
This  time  will  be  positive  for  the  forecast  images,  and  negative  for  past  weather  images.  Actual 
time  is  displayed  beneath  the  relative  forecast  time  display. 
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The  font  will  be  white  when  the  images  are  looping  and  will  turn  red  when  the  loop  has  been 
stopped. 


Relative 

Time 


FIGURE  8.  RELATIVE  FORECAST  TIME 


3.2.7  Range  Indicators. 

A  status  bar  at  the  top  of  the  weather  depiction  display  window  indicates  the  range  selected 
(i.e.  TRACON  or  200  nm)  as  well  as  the  type  of  weather  depiction  (past,  current  or  forecast). 

4.  DEMONSTRATION  APPROACH. 

The  2000  Demonstration  addressed  both  positive  and  negative  aspects  of  TCWF  utility  and  data 
presentation.  The  demonstration  also  assessed  how  well  TCWF  met  the  job  task  and  information 
requirements  of  the  Memphis  users.  Results  and  collected  data  will  be  used  to  assist  MIT/LL  in 
improving  both  the  display  and  performance  of  the  TCWF.  Additionally,  results  will  be 
forwarded  to  the  ITWS  Program  Office  within  the  FAA  for  decisions  regarding  integration  of  the 
TCWF  onto  the  ITWS  platform. 

Prior  to  demonstration  conduct,  ACB-630  evaluators  collected  baseline  data  at  both  the  ZME 
TMU  and  MEM  TRACON  before  introduction  of  the  TCWF  in  their  respective  operations  areas. 
This  was  to  ensure  baseline  data  would  not  be  confounded  by  the  presence  of  a  new  convective 
weather  information  system.  The  purpose  of  this  baseline  effort  was  to  ascertain  current 
procedures  and  weather  information  sources  used  during  convective  weather  events.  Evaluators 
also  administered  workload  scales  to  subjectively  measure  users  perceptions  of  mental  workload 
while  operating  under  convective  conditions.  Workload  was  administered  again  at  the  end  of  the 
Phase  2  data  collection  effort. 

The  purpose  of  Phase  1  data  collection  was  to  capture  information  regarding  the  utility,  ease  of 
use,  and  type  of  use  of  the  TCWF  during  line-storm  convective  weather  at  Memphis.  Phase  2 
data  collection  focused  on  collecting  feedback  for  both  air-mass  and  line-storm  convective 
weather.  The  purpose  of  using  a  phased  approach  was  to  capture  differences  in  product 
performance  and  utility  across  different  types  of  convective  weather. 

The  FAA  Technical  Center  (ACB-630)  and  MIT/LL  conducted  the  TCWF  Demonstration.  Data 
collection  took  place  at  the  demonstration  sites  listed  in  paragraph  4.4.  Participants  included: 

a.  Traffic  Management  Supervisors  (Sups),  and 

b.  TMCs 
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ACB-630  test  personnel  conducted  the  TCWF  Demonstration.  Participants  utilized  the  TCWF 
during  convective  weather  events.  Based  on  TCWF  operational  use,  evaluators  assessed  TCWF 
using  various  demonstration  techniques  described  in  section  4.2. 

4.1  DEMONSTRATION  OBJECTIVES. 

The  primary  objectives  of  the  demonstration  were: 

a.  To  determine  the  overall  utility  of  the  TCWF  including: 

1.  Incremental  forecasts  out  to  60  minutes,  and 

2.  Scoring  methodology. 

b.  To  determine  the  ease  of  use  of  current  display  concepts,  more  specifically: 

1.  Overlay  presentation, 

2.  Color  coding, 

3.  Animation,  and 

4.  Product  presentation  on  an  ITWS-type  SD  using  Unix  based  GUIs. 

c.  To  determine  if  the  TCWF  enhances  the  users’  ability  to  make  informed  and  timely 
operational  decisions  in  determining  whether  or  not  to  perform  specific  tasks  (e.g., 
open/close  approach  gates,  meter  traffic)  during  convective  weather. 

4.1.1  Definitions  of  Measures. 


This  paragraph  will  define  important  terms  used  in  the  description  of  demonstration  objectives. 
The  terms  are  as  follows: 

a.  Utility  -  This  refers  to  how  useful  the  TCWF  is  in  completing  job  tasks  related  to 
traffic  planning  as  impacted  by  convective  weather.  For  this  demonstration,  utility 
was  measured  subjectively  according  to  questionnaire  ratings.  A  user  could 
consider  a  feature  useful  if  it: 

1.  Improves  his/her  situational  awareness, 

2.  Positively  impacts  his/her  decision  making,  and/or 

3.  Positively  impacts  his/her  workload. 

It  is  important  to  keep  in  mind  that  utility  and  ease  of  use  are  distinctly  different.  A  product  may 
be  very  easy  to  use,  but  the  product  might  not  provide  a  user  with  useful  information. 

b.  Ease  of  Use  This  refers  to.  how  easy  the  forecast  is  to  use  in  completing  job  tasks 
(i.e.,  navigation,  button  use).  A  user  may  consider  a  feature  easy  to  use  if  he/she 
can  access  information  on  the  display  with  little  difficulty  and  can  easily 
discriminate  display  characters  and  symbols.  As  mentioned  above,  ease  of  use  and 
utility  are  different.  A  product  may  be  very  useful,  but  may  be  very  difficult  to  use. 
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c.  Readability  -  This  refers  to  the  extent  the  information  displayed  is  easy  to  see, 
readable,  meaningful,  and  understandable.  For  example,  overlaid  text  may  be  too 
small  to  read  or  obscured  by  other  overlays. 

d-  Situational  Awareness  -  Situational  Awareness  (SA)  is  generally  known  as  the 
perception  of  elements  in  the  environment,  an  understanding  of  what  these  elements 
mean,  and  how  they  will  affect  the  near  future.  In  order  to  maintain  SA,  it  is 
important  that  users  perceive  and  understand  elements  in  the  present  situation  and 
use  this  information  to  anticipate  future  events.  The  extent  that  operators  perceived 
the  effects  of  the  TCWF  on  SA  was  addressed. 

4.2  DEMONSTRATION  MF.TRTrS 

The  objectives  of  the  TCWF  Demonstration  were  achieved  by  obtaining  feedback  from  air  traffic 
users.  This  section  will  describe  the  demonstration  techniques  and  measurement  tools.  User 
data  collection  incorporated  the  following  techniques: 

a.  Questionnaires, 

b.  User  interviews,  and 

c.  Workload  scales. 

4.2.1  Questionnaires. 

Questionnaires  were  administered  at  the  end  of  Phases  1  and  2.  Using  a  5-point  Likert  scale, 
demonstration  participants  rated  various  aspects  of  the  TCWF  based  on  utility,  operational 
suitability,  and  frequency  of  use  as  related  to  specific  job  tasks.  Participants  also  rated  the 
product  functions  and  components  on  ease  of  use,  utility,  and  readability  in  order  to  identify 
good  interface  concepts  for  future  iterations  of  the  display.  The  questionnaire  also  included 
open-ended  questions  that  allowed  users  to  identify  aspects  of  the  TCWF  they  felt  needed 
improvement.  Given  the  differences  in  air  traffic  tasks  for  each  position  and  location 
(i.e.,  ARTCC  vs.  TRACON),  separate  questionnaires  were  developed  for  each  user  group  and 
location.  The  questionnaires  were  very  similar  in  nature  to  allow  evaluators  to  create 
comparisons  between  groups;  however,  the  uniqueness  of  each  questionnaire  enabled  evaluators 
to  identify  specific  user  needs.  Questionnaires  are  located  in  appendix  A. 

4.2.2  Interviews. 


Interview  questions  were  designed  to  obtain  user  impressions  on  the  product  in  terms  of  task 
benefit,  utility,  product  performance,  benefit  beyond  ITWS,  and  suggested  enhancements  to 
better  meet  user  requirements.  Interviews  were  conducted  with  individual  users  from  both  MEM 

and  ZME  during  both  Phases  1  and  2.  Structured  interview  questions  and  responses  are  located 
in  appendix  B. 
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4.2.3  Workload  Scales. 


The  Modified  Cooper  Harper  (MCH)  mental  workload  scale  was  used  to  measure  overall 
perceived  user  mental  workload  as  it  applied  to  operational  use  of  the  TCWF  display.  The  MCH 
is  a  unidimensional  subjective  workload  scale  which  measures  mental  workload  associated  with 
a  variety  of  perceptual,  cognitive,  and  communications  tasks.  The  MCH  is  a  sensitive  measure 
and  is  widely  applicable  to  a  variety  of  operator  tasks.  Essentially,  the  scale  is  a  decision  tree 
that  leads  the  user  to  one  of  10  ordinal  ratings  (O’Brien  &  Charlton  (eds.)  1996). 

During  baseline  data  collection,  the  MCH  was  administered  to  capture  current  perceptions  of 
mental  workload  associated  with  significant  convective  weather  events  actually  or  potentially 
impacting  air  traffic  operations.  Affected  task  areas  were  identified  and  verified  by  users.  MEM 
TRACON  tasks,  for  example,  included  changing  runways,  and  closing  arrival  gates.  Tasking  for 
ZME  TMC  users  included  decisions  affecting  Miles-In-Trail  (MIT),  and  determining  holding 
areas.  At  the  end  of  the  final  demonstration  phase,  the  MCH  was  administered  to  the  same  users 
who  initially  completed  the  MCH  during  the  baseline  phase.  Comparisons  of  both  iterations  (pre 
and  post)  of  the  MCH  were  compared  to  see  if  perceived  mental  workload  increased,  decreased, 
or  remained  the  same  after  introduction  of  the  TCWF  in  the  operational  environment.  A  sample 
workload  scale  for  one  of  the  MEM  TRACON  tasks  is  located  in  appendix  C.  Remaining  tasks 
used  an  equivalent  scale  and  are  identified  in  section  6.4. 

4.3  DATA  ANALYSIS. 

Data  was  summarized  and  tabulated.  Questionnaire  and  workload  ratings  were  analyzed  using 
descriptive  statistics.  Structured  interview  responses  were  summarized.  Results  are  presented  in 
tabular  and  graphical  formats  in  section  6.  Since  ranking  data  was  ordinal  and  a  normal 
distribution  could  not  be  assumed,  the  median  was  used  as  the  measure  of  central  tendency. 

4.4  DEMONSTRATION  SITES. 

TCWF  displays  were  located  in  both  non-operational  and  operational  settings.  MIT/LL  and 
FAA  personnel  had  access  to  the  TCWF  in  non-operational  settings.  Non-operational  settings 
included: 

a.  MIT/LL;  Lexington,  MA  and 

b.  FAA  William  J.  Hughes  Technical  Center,  Atlantic  City,  NJ. 

The  users  utilized  the  TCWF  in  operational  settings  during  convective  weather.  The 
demonstration  took  place  at  the  following  operational  sites: 

a.  FAA  MEM  TRACON;  and 

b.  FAA  ZMETMU. 
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5.  DEMONSTRATION  MANAGEMF.NT. 


ACB-630,  working  as  a  member  of  the  Weather  Sensors  and  Aviation  Weather  Research 
Integrated  Product  Team  (IPT)  (AUA-430),  and  in  conjunction  with  MIT/LL,  had  overall 
responsibility  for  the  conduct  and  coordination  of  the  TCWF  Demonstration. 

The  principle  organizations  participating  in  the  demonstration  included: 

a.  AUA-430,  FAA  Headquarters,  Washington,  DC; 

b.  ACB-630; 

c.  MIT/LL; 

d.  MEM  TRACON;  and 

e.  ZME. 

6.  RESULTS. 

Results  from  data  collected  during  TCWF  Demonstration  Phases  1  and  2  will  be  addressed 
separately  in  this  section. 

6.1  FACTORS  EFFECTING  RESULTS. 

Before  and  during  the  demonstration  period,  some  potential  issues  arose  that  may  have  impacted 
the  results  and  should  be  taken  into  consideration  when  reviewing  this  report.  Therefore, 
conclusions  derived  from  these  results  should  be  weighed  carefully.  Factors  effecting  results 
included  the  following: 

Lack  of  Convective  Weather  -  Operationally  significant  convective  weather  events  were 
infrequent.  Due  to  lack  of  convective  weather,  infrequency  of  use  was  reported  by  a  number  of 
demonstration  participants  resulting  in  fewer  opportunities  to  fully  utilize  the  TCWF.  For 
demonstration  purposes,  more  opportunities  for  TCWF  use  would  have  provided  better  insight 
into  user  perceptions  of  the  product. 

Training  Bias  -  TCWF  training  was  conducted  at  the  end  of  March  2000  by  MIT/LL  personnel  to 
all  TCWF  Memphis  participants  prior  to  TCWF  Demonstration  conduct.  Some  of  the  training 
material  included  positive  results  obtained  from  the  TCWF  1999  Demonstration,  as  well  as  data 
on  forecast  accuracy  performance  from  Dallas  and  New  York.  Dallas  TRACON  user  comments 
were  taken  out  of  context  and  did  not  include  any  negative  feedback,  i.e.  "It  (TCWF)  is  very 
useful"  and  "will  be  beneficial  to  pilots".  If  Memphis  sites  were  assumed  to  be  "clean"  (no 
previous  exposure  to  TCWF),  then  introduction  of  this  data  violated  that  assumption.  Users  may 
have  been  favorably  biased  in  their  perception  of  the  TCWF  prior  to  its  operational  availability. 
If  so,  TCWF  Demonstration  results  may  reflect  this  bias. 
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6.2  STRUCTURED  INTERVIEW  RESULTS. 


6.2.1  Phase  1  Interview  Results. 

Users  were  interviewed  during  the  Phase  1  data  collection  period  using  the  structured  interview 
questions  located  in  appendix  B.  Eight  users  (out  of  total  number  of  15  TMCs)  were  interviewed 
at  the  ZME  TMU  and  five  users  (out  of  total  of  10  Supervisors)  at  the  MEM  TRACON.  User 
responses  are  summarized  for  each  question.  Additionally,  a  table  is  provided  for  each  question 
to  provide  an  overall  summary  regarding  the  number  of  users  who  responded  with  a  positive, 
negative,  or  neutral  response.  Some  descriptive  words/phrases  are  used  for  clarity.  Phase  1 
interview  responses  for  each  user  are  summarized  in  appendix  B. 

1.  What  is  your  overall  impression  of  the  TCWF? 

MEM  TRACON  users  were  positive  overall.  Many  found  the  TCWF  very  useful  as  a  source  for 
terminal  convective  weather  information. 

Generally,  ZME  users’  overall  impression  of  the  TCWF  product  was  favorable.  Many  used  the 
TCWF  for  MEM  arrival  strategies,  rerouting,  and  flow  management.  Users  reported  the  forecast 
component  and  animated  storm  movement  capability  as  the  most  desirable  attributes  of  the 
product.  Conversely,  one  user  reported  not  using  the  product  due  to  poor  accuracy  scores, 
lowering  expectations  of  the  product’s  forecast  capability.  Table  1  provides  overall  TCWF 
impressions  from  each  user  group. 


TABLET  OVERALL  TCWF  IMPRESSION 


USER  GROUP 

POSITIVE 

NEGATIVE 

NO  OPINION 

ALL  USERS  (N=13) 

12 

1 

0 

MEM  Sups  (N=5) 

5 

0 

0 

ZME  TMCs  (N=8) 

7 

1 

0 

2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those  tasks  and  how 
did  the  TCWF  help? 

All  MEM  TRACON  users  reported  that  the  TCWF  helped  in  performing  specific  tasks  listed  in 
table  2.  The  TCWF  helped  by  predicting  where  gates,  holding  areas,  and  runways  would  be 
impacted  by  convection,  allowing  MEM  users  to  plan  accordingly. 

Five  of  the  ZME  users  reported  the  TCWF  helped  in  performing  specific  tasks.  Most  noted  are 
listed  in  table  2.  In  these  instances  the  TCWF  animated  storm  movement  and  forecast 
capabilities  were  most  helpful  in  task  areas  like  planning  gate  closures  or  anticipating  traffic 
flows.  One  user  noted  the  graphic  display  and  animation  were  very  useful  components  for 
visualizing  weather  movement.  Another  ZME  user  noted  increased  shared  situational  awareness 
with  MEM  approach  control,  stating  "we  were  both  looking  at  the  same  picture  and  could  focus 
on  the  same  view." 
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One  ZME  TMC  was  reluctant  to  use  the  TCWF  based  on  its  experimental  status.  Another  ZME 
user  reported  never  using  the  TCWF  since  ITWS  was  adequate. 


TABLE  2.  TCWF  TASK  BENEFIT 


USER  GROUP 

YES 

TASK  BENEFITS 

NO 

NO 

OPINION 

ALL  USERS  (N=13) 

10 

See  Below. 

2 

1 

MEM  Sups  (N=5) 

5 

a.  Flow  control. 

b.  Holding. 

c.  Terminal  routing. 

d.  Runway  reconfigurations, 
e  Determining  Airport 

Arrival  Rate  (AAR). 

0 

0 

ZME  TMCs  (N=8) 

5 

a.  Discontinue  holding. 

b.  Rerouting. 

c.  Gate  impacts. 

d.  Anticipating  AAR. 

2 

1 

3.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  hinder  them  ? 

All  MEM  TRACON  users  reported  no  hindrances  in  using  the  TCWF  product.  One  noted  that 
for  air-mass  convection,  product  use  was  infrequent  since  the  product  did  not  perform  well  under 
air-mass  conditions. 

All  of  the  Memphis  ARTCC  users  found  no  negative  impacts  in  using  the  TCWF.  One  user 
noted  that  by  not  using  the  product,  there  could  be  no  negative  impacts.  Table  3  provides  a 
summary  of  responses  regarding  negative  task  impacts. 


TABLE  3.  TASK  IMPACTS 


USER  GROUP 

YES 

TASK  HINDERED 

NO 

NO 

OPINION 

ALL  USERS  (N=  13) 

0 

None 

13 

0 

MEM  Sups  (N=5) 

0 

None 

5 

0 

ZME  TMCs  (N=8) 

0 

None 

8 

0 

4.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the  TCWF  have  been 
more  helpful?  If  no,  why  is  ITWS  more  beneficial? 

All  MEM  TRACON  participants  reported  benefit  beyond  ITWS.  Two  respondents  noted  that  the 
TCWF  gives  a  better  visual  weather  depiction  than  ITWS  and  that  the  TCWF  forecast  and  storm 
movement  components  were  beneficial.  Table  4  summarizes  responses  regarding  TCWF  benefit 
beyond  ITWS  along  with  helpful  TCWF  features. 
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Almost  all  (7)  ZME  users  found  TCWF  provided  benefit  beyond  ITWS.  TCWF  enabled  them  to 
better  time  weather  movement  and  ascertain  how  and  when  the  MEM  terminal  area  would  be 
affected.  TCWF  features  found  to  be  most  helpful  included  the  forecast  component  and  storm 
movement  projections  or  animated  forecast  projections.  It  was  noted  that  the  product  was  mostly 
used  for  the  terminal  area,  although  a  200  nm  projection  was  available.  One  respondent  noted 
that  MEM  only  effects  20%  of  their  operations;  therefore,  the  TCWF  did  not  fulfill  requirements 
for  a  majority  of  operations  within  ZME  airspace.  One  respondent  noted  no  benefit  beyond 
ITWS  since  the  TCWF  forecast  was  inaccurate. 


TABLE  4.  TCWF  BENEFIT  BEYOND  ITWS 


USER  GROUP 

YES 

HELPFUL  TCWF 
FEATURES 

NO 

NO 

OPINION 

ALL  USERS  (N=13) 

12 

See  Below. 

1 

0 

MEM  Sups  (N=5) 

5 

a.  Forecast. 

b.  Storm  movement. 

c.  Graphical  weather 
depiction. 

0 

0 

ZME  TMCs  (N=8) 

7 

a.  Forecast. 

b.  Storm  movement. 

1 

5.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or  out  of  your 
( MEM  or  ZME)  airspace  during  convective  weather?  If  yes,  what  information  does  the 
TCWF  provide  that  helps? 

All  MEM  TRACON  participants  reported  the  TCWF  enabled  them  to  more  efficiently  handle  the 
flow  of  traffic  into  or  out  of  their  airspace  during  convective  weather.  Users  noted  helpful 
TCWF  information  included  the  30-and  60-minute  forecasts  and  forecast  accuracy  scores. 

Seven  ZME  participants  found  TCWF  helpful  in  handling  traffic  flow  during  convective  weather 
events  as  shown  in  table  5.  Again,  it  was  noted  that  the  greatest  benefit  was  to  anticipate  MEM 
terminal  impacts.  There  was  little  to  no  perceived  benefit  for  the  rest  of  ZME  airspace. 


TABLE  5.  TRAFFIC  FLOW  BENEFIT 


USER  GROUP 

YES 

HELPFUL  TCWF 
INFORMATION 

NO 

NO 

OPINION 

ALL  USERS  (N=13) 

12 

See  Below. 

0 

1 

MEM  Sups  (N=5) 

5 

a.  Forecast. 

b.  Accuracy  scores. 

0 

0 

ZME  TMCs  (N=8) 

7 

a.  Terminal  weather. 

b.  Forecast. 

0 

1 

6.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 
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Suggestions  for  improvement  from  both  MEM  TRACON  and  ZME  participants  are  summarized 
in  table  6.  Some  MEM  users  did  not  want  to  combine  the  TCWF  with  the  ITWS  platform,  but 
leave  as  a  stand-alone  display.  Users  explained  that  this  would  make  TCWF  access  less 
cumbersome  and  reduce  display  clutter.  Given  the  integrated  option,  one  TMC  suggested  testing 
the  TCWF/ITWS  product  before  operational  implementation. 

Two  MEM  TRACON  users  indicated  that  training  was  insufficient  to  fully  utilize  the  TCWF 
operationally.  One  user  found  the  forecast  color  intensity  (yellow  and  yellow-hatched 
presentations)  to  be  confusing  and  suggested  using  standard  ITWS  level  weather  intensity  colors 
(i.e.  the  color  red  denoting  the  highest  intensities). 

One  ZME  user  suggested  improving  the  forecast  presentation  by  indicating,  by  text  or  some 
other  indicator,  when  an  area  of  airspace  would  be  clear  or  when  weather  would  impact  it  (i.e., 
storms  will  be  north  of  the  airport  for  the  next  20  minutes).  Two  TMCs  noted  the  forecast 
accuracy  should  be  improved.  Another  suggested  a  larger  display  would  improve  readability  of 
the  display. 


TABLE  6.  SUGGESTED  TCWF  MODIFICATIONS 


USER  GROUP 

MODIFICATIONS/IMPROVEMENTS 

MEM  Sups  (N=5) 

a.  Keep  separate  from  ITWS. 

b.  Include  gust  front  and  wind  information. 

c.  Integrate  with  ITWS  as  a  separate  window. 

d.  Improve  TCWF  training. 

e.  Forecast  weather  intensity  level  colors  should  be 
the  same  as  those  used  on  ITWS. 

ZME  TMCs  (N=8) 

a.  Improve  growth  and  decay  capability. 

b.  Improve  forecast  accuracy. 

c.  Integrate  with  ITWS. 

d.  Include  a  ZME  boundary  map  overlay. 

e.  Increase  product  range  to  include  all  ZME  airspace. 

f.  Larger  display  monitor. 

6.2.2  Phase  2  Interview  Results. 

Participants  were  interviewed  at  both  MEM  and  ZME  sites.  These  included  five  MEM 
TRACON  Sups  and  twelve  ZME  TMCs. 

1.  What  is  your  overall  impression  of  the  product?  Has  that  impression  changed  since  mid- 
July?  If  so,  how? 

Consistent  with  Phase  1  results,  overall  impressions  of  the  TCWF  product  for  most  users  from 
both  MEM  TRACON  and  the  ZME  TMU  were  positive.  See  table  7  for  a  summary  of 
comments. 
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TABLE  7.  TCWF  OVERALL  IMPRESSION 


USER  GROUP 

POSITIVE 

NEGATIVE 

NO  OPINION 

ALL  USERS  (N=17) 

15 

1 

1 

MEM  Sups  (N=5) 

4 

0 

1 

ZME  TMCs  (N=12) 

11 

1 

0 

Responses  included  words/phrases  such  as:  "great",  "good  product",  "like  the  product",  and 
"impressed  with  the  product".  One  ZME  TMC  noted  that  although  favorably  impressed  by  the 
TCWF,  confidence  in  its  accuracy  during  air-mass  convection  decreased. 

One  ZME  user  responded  negatively,  stating  there  was  no  use  for  the  TCWF  since  it  only 
focused  on  one  small  area  of  ZME  airspace  and  was  mostly  used  for  anticipating  runway 
impacts. 

One  MEM  TRACON  user  had  no  opinion,  explaining  convective  weather  events  had  been  too 
infrequent  to  fully  utilize  the  product. 

2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those  tasks  and  how 
did  the  TCWF  help? 

All  MEM  TRACON  users  reported  some  TCWF  task  benefit  and  reported  it  helpful  for  general 
planning  purposes.  TCWF  aided  planning  by  forecasting  where  convective  weather  would  be 
and  when  it  would  move  into  or  leave  an  area  (i.e.  fix,  runway,  airport).  The  TCWF  reportedly 
benefited  several  tasks  (see  table  8)  such  as  opening/closing  gates,  and  changing  runway 
configurations. 

Most  ZME  users  (10)  also  indicated  the  TCWF  provided  task  benefit.  Overall  it  was  considered 
a  good  planning  tool  by  providing  a  general  indication  of  when  weather  would  impact  MEM. 
The  TCWF  reportedly  benefited  specific  tasks  (see  table  8)  such  as  traffic  flow  planning  based 
on  MEM  airport  weather  impacts,  and  arrival  and  departure  gate  impacts. 


TABLE  8.  TCWF  TASK  BENEFIT 


USER  GROUP 

YES 

TASKS  BENEFITED  ~~~ 1 

NO 

NO  OPINION 

ALL  USERS  (N=17) 

15 

See  Below. 

0 

2 

MEM  Sups  (N=5) 

5 

a.  Open/close  gates/fixes. 

b.  Change  runway  configuration. 

c.  Determine  holding  at  fixes. 

d.  Anticipate  MIT. 

e.  Plan  traffic  flow/reroutes. 

f.  Open/close  airport. 

g.  Improved  SA  with  ZME. 

0 

0 

ZMETMCs  (N=12) 

10 

a.  Ascertain  weather  impact 
on  MEM. 

b.  Plan  arrival  and  departure 
gate  impacts. 

c.  Determine  holds. 

d.  Plan  traffic  flow. 

e.  Anticipate  MIT 

f.  Shared  SA  with  MEM 
TRACON. 

0 

2 

3.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or  out  of  MEM 
or  ZME  airspace  during  convective  weather?  If  yes,  what  information  does  the  TCWF 
provide  that  helps? 

All  MEM  TRACON  users  reported  the  TCWF  enabled  them  to  more  efficiently  handle  the  flow 
of  traffic  into  and  out  of  MEM  airspace.  Information  reported  to  be  most  helpful  included  the 
incremental  forecast  and  storm  movement  projections 

Most  ZME  TMC  users  (10)  reported  TCWF  benefit  in  handling  traffic  flow  around  MEM 
airspace,  but  not  all  of  ZME  airspace.  Two  users  had  no  opinion.  Similar  to  MEM  responses, 
ZME  users  found  the  forecast  and  forecast  weather  movement  (animation)  components  most 
helpful.  See  table  9  for  user  response  summaries. 


TABLE  9.  TRAFFIC  FLOW  BENEFIT 


USER  GROUP 

YES 

HELPFUL  TCWF 
INFORMATION 

NO 

NO  OPINION 

ALL  USERS  (N=17) 

15 

See  Below. 

0 

2 

MEM  Sups  (N=5) 

5 

a.  Incremental  forecast. 

b.  Weather  movement. 

0 

0 

ZMETMCs  (N=12) 

10 

a.  Incremental  forecast. 

b.  Weather  movement. 

0 

2 

4.  Do  you  recall  any  convective  weather  scenarios  that  had  impacted  MEM/ZME  airspace?  If 
so,  was  the  TCWF  used  for  this  weather  event?  For  what  tasks? 


Many  users  recalled  specific  or  general  instances  of  using  the  TCWF  in  performance  of  some 
tasks.  All  recollections  of  TCWF  performance  were  positive  and  are  summarized  in  table  10. 

Two  MEM  users  recalled  TCWF  use  during  line-storm  convective  weather,  particularly  for 
holding  situations  (i.e.  when  to  take  aircraft  out  of  holding).  One  MEM  user  reported  using  the 
TCWF  for  making  runway  changes,  determining  flow  rates  and  anticipating  arrival  and  departure 
impacts.  Additional  information  sources  included  ITWS,  ASR-9  radar,  and  the  Aircraft 
Situation  Display  (ASD).  Two  MEM  users  had  no  specific  recollection. 

Four  ZME  users  reported  TCWF  use  during  line-storm  convection.  Two  TMCs  made  comments 
based  on  use  during  general  convective  weather  situations.  The  TCWF  was  reportedly  accurate 
and  helpful  in  performance  of  many  tasks,  including  traffic  flow  management,  anticipating 
arrival/departure  gate,  and  MEM  airport  impacts  as  well  as  anticipating  MIT  restrictions  and 
determining  when  to  take  aircraft  out  of  holding.  Other  weather  information  sources  included 
the  CWSU,  WARP  Briefing  Terminal,  and  ITWS.  Four  ZME  users  had  no  specific  recollections 
of  TCWF  use. 


TABLE  10.  TCWF  USE  DURING  CONVECTIVE  WEATHER  EVENTS 


1  USER 
GROUP 

CONVECTIVE  WX  EVENT 

TCWF  USE 

OTHER 

WX 

INFOR 

MEM 

Sups 

(N=3) 

Yesterday  a  line  (of 
thunderstorms)  passed  through. 
Was  about  300-400  miles  long. 

Was  able  to  predict  when  aircraft  could 
come  out  of  holding.  Nailed  forecast 
with  only  1  aircraft  having  to  divert. 
TCWF  used  for  about  70-80%  of 
planning. 

rrws  for 

intensity 

levels. 

ASR-9 

radar. 

Used  3-4  times  with  line-storm 
situations. 

Used  to  determine  going  into  holdings 
and  coming  out,  and  changing  runway 
configurations. 

ASD  j 

Fed-ex 

weather 

conference 

call. 

General  convective  events. 

Used  TCWF  for  runway  changes,  flow 
rates,  anticipating  impacts  to  arrivals 
and  departures,  and  being  able  to 
predict  where  weather  will  be. 

Used  TCWF  in  conjunction  with  other 
products. 

rrws  SEP  | 

ZME 

TMCs 

(N=6) 

Last  night  (11/6)  had  a  line  of 
thunderstorms  from  about  100 
miles  south  of  MEM  to  about  80 
miles  north  of  MEM. 

Used  to  determine  miles-in-trail  and 
holding.  Was  able  to  spread  aircraft  out 
to  make  a  manageable  situation;  held 
aircraft  and  brought  them  in  on  the  back 
side  of  the  line;  and  able  to  minimize 
holding  time  and  bring  aircraft  in  as 
soon  as  weather  cleared  MEM.  Product 
had  good  accuracy  in  this  event. 
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TABLE  10.  TCWF  USE  DURING  CONVECTIVE  WEATHER  EVENTS  (Continued) 


USER 

GROUP 

CONVECTIVE  WX  EVENT 

TCWF  USE 

OTHER 

WX 

INFOR 

Line  of  storms  on  1 1/6. 

Yesterday’s  weather  situation  was  a 
good  example  of  how  TCWF  has  been 
used.  Was  able  to  predict  approximate 
times  to  move  aircraft  to  gates;  did  not 
have  to  divert  because  of  TCWF 
information. 

Line  of  storms  on  1 1/6. 

Forecasted  accurately  by  TCWF,  which 
was  only  off  by  10  minutes  with  airport 
arrivals.  Due  to  TCWF  information, 
knew  aircraft  had  to  be  moved  north 
around  the  line  and  brought  in  on  the 
backside  of  the  line.  Relied  solely  upon 
the  TCWF. 

General  convective  weather  use. 

TCWF  forecasts  are  easier  to  interpret 
than  ITWS.  Can  tell  if  airport  will  be 
impacted  and  if  MIT  and  spacing  are 
needed.  With  ITWS,  have  to  assume 
airport  will  be  impacted  due  to  the  lack 
of  forecast  information.  Line-storms 
tend  to  be  more  accurate;  isolated 
storms  are  harder  to  predict.  However, 
attention  is  paid  when  TCWF  is 
forecasting  something  with  isolated 
storms. 

ITWS 

ZME 

TMCs 

(N=6) 

Line  of  storms  on  1 1/6. 

Discussed  TCWF  output  with  CWSU  to 
see  what  they  think,  (i.e.  if  forecast 
portrayal  is  viable). 

CWSU 
WARP  -  to 
get  bigger 
picture. 

General  convective  weather  use. 

Used  TCWF  to  monitor  the  weather  and 
pass  information  to  other  Centers  to 
avoid  overloading  sectors.  Have  used 
TCWF  information  for  justification  of 
Center  decisions  to  ATCSCC. 

5.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  hinder  them  ? 


No  user  from  either  facility  (MEM  or  ZME)  reported  hindrances  to  tasking  by  using  the  TCWF 
product.  Table  1 1  provides  a  summary  of  responses  to  this  question. 
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TABLE  11.  TASK  IMPACTS 


USER  GROUP 

YES 

TASK  HINDERED 

NO 

NO 

OPINION 

ALL  USERS  (N=  17) 

0 

None 

16 

1 

MEM  Sups  (N=5) 

0 

None 

5 

0 

ZMETMCs  (N=12) 

0 

None 

11 

1 

6.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the  TCWF  have  been 
more  helpful?  If  no,  why  is  ITWS  more  beneficial? 

All  MEM  TRACON  users  reported  TCWF  benefit  beyond  ITWS.  Many  considered  the  1-hour 
animated  forecast  and  areal  storm  coverage  of  the  TCWF  an  improvement  over  the  ITWS  SEP 
20-minute  forecast  of  the  storm’s  leading  edge. 

Most  ZME  users  (9)  found  TCWF  provided  benefit  beyond  ITWS.  Again,  the  forecast  and 
forecast  portrayal  (animation  and  area  coverage)  were  considered  the  greatest  TCWF  assets. 
Two  users  reported  no  TCWF  benefit  beyond  ITWS  since  the  TCWF  does  not  include  wind 
information,  and  forecasts  are  not  always  reliable.  Table  12  provides  a  summary  of  reported 
TCWF  benefit  beyond  ITWS. 


TABLE  12.  TCWF  BENEFIT  BEYOND  ITWS 


USER  GROUP 

YES 

HELPFUL  TCWF 
FEATURES 

NO 

NO 

OPINION 

ALL  USERS  (N=  17) 

14 

See  Below. 

2 

1 

MEM  Sups  (N=5) 

5 

a.  Better/longer  forecast 
prediction  than  ITWS 
SEP. 

b.  Better  visual  forecast 
presentation  than 
ITWS. 

0 

0 

ZME  TMCs  (N=12) 

9 

a.  Forecast. 

b.  Forecast  animation. 

2 

1 

7.  Did  the  30-  and  60-minute  forecast  accuracy  scores  effect  your  decision  to  use  or  not  use  the 
TCWF? 

Three  of  the  five  MEM  TRACON  users  reported  not  using  forecast  accuracy  scores  to  base 
decisions  on  TCWF  use.  The  remaining  users  (2)  indicated  accuracy  scores  did  affect  their 
decision  to  use  or  not  use  the  TCWF  product.  Criteria  for  score  use  were  not  specific,  although 
these  users  indicated  that  if  scores  were  low,  the  product  would  not  be  relied  on.  Conversely,  if 
scores  were  higher,  forecast  accuracy  would  be  confirmed  and  likelihood  to  use  the  product 
increased. 
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Most  (7)  ZME  users  reported  using  forecast  accuracy  scores  to  determine  whether  or  not  to  use 
the  TCWF  product,  whereas  five  users  did  not.  Accuracy  score  criteria  for  product  use  included 
scores  above  40%  to  scores  above  75%.  Table  13  summarizes  user  responses  for  accuracy  score 
use  and  criteria. 


TABLE  13.  USE  OF  ACCURACY  SCORES  AND  CRITERIA  FOR  USE 


USER  GROUP 

USED 

SCORES 

WHY  USED 

DID  NOT 
USE 
SCORES 

WHY  NOT  USED 

ALL  USERS 
(N=17) 

9 

See  Below. 

8 

See  Below. 

MEM  Sups  (N=5) 

2 

a.  Would  not  rely 
on  TCWF  if  ‘ 
scores  were 

low. 

b.  Higher  scores 
would  confirm 
accuracy,  esp. 
w/  line-storms. 

3 

a.  Make  decisions 
to  use  TCWF 
based  on  forecast 
image. 

b.  Only  need  30 
min.  outlook. 
Scores 
unimportant. 

ZMETMCs  (N=12) 

7 

a.  Scores  >  40% 

b.  Scores  >  50% 

c.  Scores  >  70% 

d.  Scores  >  75% 

5 

a.  Unimportant 
during  line- 
storms. 

b.  Waste  of  space. 

8.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How?  Do  you  have  any 

comments  or  concerns  regarding  the  forecast  product? 


Suggestions  for  improvements/modifications  to  the  TCWF  as  well  as  additional  comments 
regarding  the  forecast  product  are  summarized  in  table  14. 
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TABLE  14.  SUGGESTED  TCWF  MODIFICATIONS/COMMENTS 


USER  GROUP 

MODIFICATIONS/IMPROVEMENTS 

MEM  Sups  (N=5) 

a.  Incorporate  wind  direction  and  velocity. 

b.  Improve  training. 

c.  Keep  separate  from  ITWS. 

d.  If  integrated  with  ITWS,  display  on  larger  monitor. 

e.  Make  available  to  controllers. 

ZME  TMCs  (N=12) 

a.  Integrate  TCWF  with  ITWS. 

b.  Keep  TCWF  separate  from  ITWS. 

c.  Increase  product  range. 

d.  Improve  accuracy. 

e.  Include  overlays  of  Centers,  fixes,  and  counties. 

f.  Provide  longer  (to  1  hr.)  past  history  of  weather. 

g.  Make  available  to  controllers. 

h.  Wider  map  boundary  lines  for  easier  discemability. 

i.  Effective  and  trustworthy  -  needs  to  be  at  other 
locations  (i.e.  Federal  Express). 

6.3  QUESTIONNAIRE  RANKING  RESULTS. 

This  section  will  describe  the  results  from  the  questionnaires  from  both  Memphis  facilities  for 
data  collection  Phases  1  and  2.  Results  will  be  discussed  separately  for  the  interface  and  job  task 
portions  of  the  questionnaires. 

6.3.1  Phase  1  Interface  Questions. 


Questionnaire  ranking  results  regarding  the  TCWF  interface  and  functionality  are  summarized  in 
table  15  for  all  users,  MEM  TRACON  users  only,  and  ZME  TMU  users  only  for  each  product 
feature.  For  a  definition  of  measures,  see  section  4.1.1.  Rating  scale  weights  are  defined  under 
corresponding  tables. 

In  general,  users  rated  the  product  features  favorably.  All  product  features  received  suitability 
ratings  of  moderately  suitable  to  totally  suitable  on  the  dimensions  of  utility,  readability,  and 
ease  of  use. 


TABLE  15.  MEDIAN  SCORE  PER  FEATURE 


ALL  USERS  (N 

=12) 

MEM  SUPS  (N=5) 

ZMETMCS  ( 

;n=7) 

Product 

Utility 

Read 

Ease 

Utility 

Read 

Ease 

Utility 

Read 

Ease 

Display  Window 

5 

5 

5 

5 

5 

5 

5 

4 

5 

Time  Display 

5 

5 

N/A 

5 

5 

N/A 

5 

5 

N/A 

Range  Indicators 

5 

5 

5 

5 

5 

5 

5 

5 

5 

Forecast  Color 
Scale 

5 

5 

N/A 

5 

5 

N/A 

5 

5 

N/A 

Movie  Loop 

5 

5 

5 

5 

5 

5 

5 

5 

5 

Loop  Control 

5 

5 

5 

5 

5 

5 

5 

5 

5 

Forecast 

Accuracy  Score 

4.5 

5 

N/A 

5 

5 

N/A 

4 

5 

N/A 

Overlays 

5 

5 

5 

5 

5 

5 

4 

4.5 

5 

Windowing 

Options 

5 

5 

5 

5 

5 

5 

5 

5 

5 

Status  Indicators 

kT  .  **  M  1  •  « 

5 

5 

N/A 

5 

5 

N/A 

4 

4 

N/A 

Note:  Median  values  are  reported  for  utility,  readability  (Read),  and  ease  of  Use  (Ease)  for  each  feature.  Scale: 

1  =  Totally  Unsuitable,  2  =  Moderately  Unsuitable,  3  =  Suitable  with  Modification,  4  =  Moderately  Suitable,  5  = 
Totally  Suitable. 


Table  16  summarizes  user  responses  regarding  the  OPTIMAL  FORECAST  LENGTH  required 
for  tasks  during  convective  weather.  MEM  TRACON  users  were  split  between  60  and  90- 
minute  forecast  durations.  One  preferred  a  30-minute  forecast. 

Similar  results  were  obtained  from  ZME  users  where  three  preferred  the  existing  60-minute 
forecast  and  three  a  90-minute  forecast.  One  user  preferred  the  existing  30-minute  forecast. 


TABLE  16.  FREQUENCY  OF  RESPONSES  -  OPTIMAL  FORECAST  LENGTH 


USER  GROUP 

30 

MINUTES 

60 

MINUTES 

90 

MINUTES 

120 

MINUTES 

ALL  USERS  (N=12) 

2 

5 

5 

0 

MEM  Sups  (N=5) 

1 

2 

2 

0 

ZME  TMCs  (N=7) 

1 

3 

3 

0 

Table  17  summarizes  user  responses  regarding  the  amount  of  time  the  TCWF  is  viewed  from  the 
50  NM  (TRACON)  RANGE.  Mean  percentage  of  time  MEM  TRACON  users  viewed  the 
TCWF  at  50  nm  was  69%.  Mean  viewing  time  for  ZME  users  and  all  users  was  68%. 
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TABLE  17.  MEAN  PERCENTAGE  -  TIME  THE  50  NM  RANGE  IS  VIEWED 


USER  GROUP 

% 

ALL  USERS  (N=12) 

68 

MEM  Sups  (N=5) 

69 

ZME  TMCs  (N=7) 

68  : 

Table  18  summarizes  user  responses  regarding  the  amount  of  time  the  TCWF  is  viewed  from  the 
200  NM  RANGE.  MEM  TRACON  users  indicated  the  200  nm  range  was  viewed  an  average  of 
31%  of  the  time.  Mean  percentage  time  ZME  users  viewed  the  200  nm  range  was  somewhat 
higher,  at  38%.  Mean  viewing  time  for  all  users  was  35%. 

TABLE  18.  MEAN  PERCENTAGE  -  TIME  200  NM  RANGE  IS  VIEWED 


USER  GROUP 

% 

ALL  USERS  (N=12) 

35 

MEM  Sups  (N=5) 

31 

ZME  TMCs  (N=7) 

38 

Table  19  summarizes  user  responses  regarding  their  need  for  a  RANGE  GREATER  THAN 
200  NM.  Seven  users  indicated  the  200  nm  range  was  sufficient;  however,  ZME  users  preferred 
a  300  to  450  nm  range  to  obtain  a  better  view  of  total  ZME  airspace.  One  TRACON  user 
indicated  a  range  beyond  200  nm  would  be  optimal,  however,  the  optimal  range  was  not 
reported. 


TABLE  19.  FREQUENCY  OF  RESPONSES  -  RANGE  GREATER  THAN  200  NM 


USER  GROUP 

YES 

OPTIMAL 

RANGE 

NO 

ALL  USERS  (N=12) 

5 

See  Below. 

7 

MEM  Sups  (N=5) 

1 

N/A 

4 

ZME  TMCs  (N=7) 

4 

300  nm  - 
450  nm 

3 

Table  20  summarizes  user  responses  regarding  perceptions  on  the  accuracy  of  the  FORECAST 
ACCURACY  SCORE.  The  majority  of  users  responded  with  "yes",  the  forecast  accuracy  score 
matched  their  impressions  of  the  actual  forecast.  Two  ZME  participants  had  no  opinion. 
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TABLE  20.  FREQUENCY  OF  RESPONSES  -  FORECAST  ACCURACY  SCORE 


USER  GROUP 

YES 

NO 

N/A 

ALL  USERS  (N=12) 

10 

0 

2 

MEM  Sups  (N=5) 

5 

0 

0 

ZME  TMCs  (N=7) 

5 

0 

2 

6.3.2  Phase  2  Interface  Questions. 

Phase  2  questionnaire  ranking  results  regarding  the  TCWF  interface  and  functionality  are 
summarized  in  table  21  for  all  users,  MEM  TRACON  users  only,  and  ZME  TMU  users  only  for 
each  product  feature.  In  general,  users  rated  the  product  features  favorably.  All  product  features 
received  suitability  ratings  of  moderately  suitable  to  totally  suitable  on  the  dimensions  of  utility, 
readability  and  ease  of  use. 


TABLE  21.  MEDIAN  SCORE  PER  FEATURE 


ALL  USERS  (N=16) 

MEM  SUPS  (N=5) 

ZME1 

[MCS  (1 

N=ll) 

Product 

Utility 

Read 

Ease 

Utility 

Read 

Ease 

Utility 

Read 

Ease 

Display  Window 

5 

5 

4.5 

5 

5 

4 

5 

5 

5 

Time  Display 

5 

5 

N/A 

5 

5 

N/A 

5 

5 

N/A 

Forecast  Color 
Scale 

5 

5 

N/A 

5 

5 

N/A 

5 

5 

N/A 

Movie  Loop 

5 

4 

4 

4 

4 

4 

5 

5 

5 

Loop  Control 

4 

5 

4 

4 

4 

5 

5 

5 

Forecast 

5 

5 

N/A 

5 

5 

N/A 

5 

4 

N/A 

Accuracy  Score 

Overlays 

5 

5 

5 

5 

5 

5 

5 

5 

5 

Windowing 

Options 

5 

5 

5 

5 

■ 

H 

5 

5 

5 

Status  Indicators 

5 

4.5 

N/A 

5 

5 

N/A 

5 

5 

N/A 

Note:  Median  values  are  reported  for  utility,  readability  (Read),  and  ease  of  Use  (Ease)  for  each  feature.  Scale: 
1  =  Totally  Unsuitable,  2  =  Moderately  Unsuitable,  3  =  Suitable  with  Modification,  4  =  Moderately  Suitable, 

5  =  Totally  Suitable. 


Table  22  summarizes  user  responses  regarding  the  OPTIMAL  FORECAST  LENGTH  for  their 
tasks  during  convective  weather.  Many  users  indicated  the  available  60-minute  forecast  was  the 
optimal  forecast  duration.  MEM  TRACON  responses  were  consistent  with  Phase  1  results, 
where  four  users  found  the  existing  60-minute  forecast  optimal.  The  majority  of  ZME  users  (7) 
also  preferred  the  60-minute  forecast.  Only  one  ZME  user  requested  a  forecast  duration  out  to 
120  minutes.  The  90-minute  forecast  was  not  considered  optimal  by  any  users. 


TABLE  22.  FREQUENCY  OF  RESPONSES  -  OPTIMAL  FORECAST  LENGTH 


USER  GROUP 

30 

MINUTES 

60 

MINUTES 

90 

MINUTES 

120 

MINUTES 

ALL  USERS  (N=15) 

3 

11 

0 

1 

MEM  Sups  (N=5) 

1 

4 

0 

0 

ZME  TMCs  (N=10) 

2 

7 

0 

1 

Table  23  summarizes  user  responses  regarding  PREFERRED  VIEWING  RANGES,  or  the 
amount  of  time  the  TCWF  is  viewed  from  any  of  the  user-selected  ranges.  These  include:  5,  15, 
30,  50  (TRACON),  100  and  200  nm  ranges.  All  MEM  users  reported  50  nm  was  the  range  of 
choice.  Many  ZME  (4)  users  also  preferred  the  50  nm  range.  Three  ZME  users  preferred  the 
100  nm  range.  None  of  the  ZME  users  stated  a  preference  for  any  range  over  100  nm. 


TABLE  23.  PREFERRED  TCWF  RANGE  VIEWING 


USER  GROUP 

5  NM 

15  NM 

30  NM 

50  NM 

100  NM 

200  NM 

ALL  USERS  (N="l6) 

0 

1 

3 

9 

3 

0 

MEM  Sups  (N=5) 

0 

0 

0 

5 

0 

0 

ZME  TMCs  (N=ll) 

0 

1 

3 

4 

3 

0 

Table  24  summarizes  user  responses  regarding  their  need  for  a  RANGE  GREATER  THAN 
200  NM.  Optimal  ranges  for  eight  ZME  users  were  beyond  200  nm  (from  400  to  600  nm). 
Three  ZME  users  noted  the  current  200  nm  range  was  adequate;  whereas,  one  indicated  that 
100  nm  would  be  the  optimal  range.  One  TRACON  user  preferred  a  range  beyond  200  nm, 
however,  the  optimal  range  identified  was  an  existing  one. 


TABLE  24.  FREQUENCY  OF  RESPONSES  -  RANGE  GREATER  THAN  200  NM 


USER  GROUP 

YES 

OPTIMAL 
RANGES  IN 
NM 

NO 

ALL  USERS  (N=16) 

9 

See  Below. 

7 

MEM  Sups  (N=5) 

1 

a.  100-200 

4 

ZME  TMCs  (N=ll) 

8 

1 

1 

3 

ACCURACY  SCORE  PERCEPTIONS  for  both  the  50  and  200  nm  ranges  were  solicited.  Users 
were  asked  if  the  accuracy  scores  at  both  of  these  ranges  reflected  the  actual  forecast 
performance  of  the  TCWF.  Tables  25  and  26  summarize  these  impressions.  For  both  ranges,  all 
MEM  users  and  most  ZME  users  reported  the  accuracy  scores  matched  their  impression  of  the 
TCWF  performance  during  the  forecast  predicted. 


TABLE  25  -  DOES  ACCURACY  SCORE  MATCH  ACTUAL  FORECAST  -  50  NM 


USER  GROUP 

YES 

NO 

N/A 

ALL  USERS  (N=  16) 

15 

0 

1 

MEM  Sups  (N=5) 

5 

0 

0 

ZME  TMCs  (N=l  1) 

10 

0 

1 

TABLE  26  -  DOES  ACCURACY  SCORE  MATCH  ACTUAL  FORECAST  -  200  NM 


USER  GROUP 

YES 

NO 

N/A 

ALL  USERS  (N=16) 

14 

0 

2 

MEM  Sups  (N=5) 

4 

0 

1 

ZME  TMCs  (N=ll) 

10 

0 

1 

6.3.3  Phase  1  Task  Questions. 

Questionnaire  results  summarizing  TCWF,  ITWS  SEP,  and  CWSU  utility  for  specific  job  tasks 
during  convective  weather  are  presented  in  tables  27  and  29.  Users  were  asked  not  to  compare 
the  three  information  sources  to  each  other,  but  to  rank  each  one  separately,  as  it  applied  to  the 
task.  Ranking  results  summarizing  the  frequency  of  use  of  the  TCWF  to  complete  the  specific 
tasks  listed  are  presented  in  tables  28  and  30. 

Table  27  summarizes  the  UTILITY  SCORES  for  MEM  TRACON  users.  With  the  exception  of 
the  "change  runway  configuration"  task,  users  considered  the  TCWF  and  the  ITWS  SEP 
component  equally  useful  for  all  task  areas.  Users  did  indicate  usefulness  of  CWSU  information 
was  borderline  for  determining  departure  and  arrival  flow  rates.  This  may  be  a  function  of  the 
limited  interaction  between  MEM  TRACON  Sups  and  the  ZME  CWSU. 


TABLE  27.  MEDIAN  UTILITY  SCORE  FOR  FORECAST  PRODUCTS 

(MEM  TRACON  ONLY) 


TASKS 

TCWF 

ITWS 

SEP 

CWSU 

Change  Runway  Configuration 

4.5 

5 

4 

Determine  When  to  Close  Arrival  Gates 

5 

5 

5 

Determine  When  to  Close  Departure  Gates 

5 

5 

5 

Determine  When  to  Reopen  Departure  Gates 

5 

5 

5 

Plan  Departure  Radials 

5 

5 

5 

Determine  Departure  Flow  Rate 

5 

5 

3.5 

Determine  Arrival  Flow  Rate 

5 

5 

3.5 

Coordinate  SWAPs 

4 

4 

4 

Aid  Situational  Awareness 

5 

5 

5 

Note:  1  =  Of  No  Use,  2  =  Of  Little  Use,  3  =  Borderline,  4  =  Somewhat  Useful,  5  =  Extremely  Useful, 
and  N/A  =  Product  not  used  for  task. 


Table  28  summarizes  FREQUENCY  OF  USE  of  the  TCWF  to  complete  the  specific  MEM 
TRACON  tasks  during  convective  weather.  MEM  TRACON  users  utilized  the  TCWF  at  least 
70%  of  the  time  for  all  identified  tasks.  Users  also  indicated  that  the  TCWF  was  used  over  90% 
of  the  time  to  aid  situational  awareness. 


TABLE  28.  MEDIAN  FREQUENCY  SCORE  OF  TCWF  USE  PER  TASK 

(MEM  TRACON  ONLY) 


TASKS 

FREQUENCY 

Change  Runway  Configuration 

4 

Determine  When  to  Close  Arrival  Gates 

4 

Determine  When  to  Close  Departure  Gates 

4 

Determine  When  to  Reopen  Departure  Gates 

4 

Plan  Departure  Radials 

4 

Determine  Departure  How  Rate 

4  1 

Determine  Arrival  Flow  Rate 

4 

Coordinate  SWAPs 

4 

Aid  Situational  Awareness 

5 

Note:  1  =  Hardly  Ever  (less  than  10%),  2  =  Seldom  (about  30%),  3  =  Now  and 
Then( about  50%),  4  =  Frequently  (about  70%),  5  =  Almost  Always  (at  least  90%),  and 
N/A  =  Product  not  used  for  task. 


Table  29  summarizes  the  UTILITY  SCORES  for  ZME  TMU  users.  ZME  users  ranked  the 
TCWF  as  useful  as  the  ITWS  SEP  forecast  information  for  every  task  listed.  For  anticipating 
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Miles-In-Trail  (MIT)  restrictions,  the  TCWF  was  rated  slightly  higher  than  the  ITWS  SEP.  ZME 
TMU  users  also  rated  the  CWSU  information  as  being  useful  for  most  tasks. 


TABLE  29.  MEDIAN  UTILITY  SCORE  FOR  FORECAST  PRODUCTS  (ZME  ONLY) 


TASKS 

TCWF 

ITWS  SEP 

CWSU 

Plan  Aircraft  Re-Routes 

4 

4 

4 

Anticipate  MIT  Restrictions 

4 

3.5 

4 

Open/Close/Reconfigure  Arrival 
Transition  Areas  (ATAs) 

4 

4 

4 

Open/Close/Reconfigure  Departure 
Transition  Areas  (DTAs) 

4 

4 

4 

Determine  Holding  Areas 

4 

4 

4 

Anticipate  Airport  Closures 

4 

4 

4 

Coordinate  Military  Release  Areas 

2 

2 

2 

Ad  Situational  Awareness 

5 

5 

4.5 

Note:  1  =  Of  No  Use,  2  =  Of  Little  Use,  3  =  Borderline,  4  =  Somewhat  Useful,  5  =  Extremely  Useful, 
and  N/A  =  Product  not  used  for  task. 


Table  30  summarizes  FREQUENCY  OF  USE  of  the  TCWF  to  complete  the  specific  ZME  TMU 
tasks  during  convective  weather.  ZME  users  utilized  the  TCWF  most  frequently  for  aiding 
situational  awareness.  The  TCWF  was  used  at  least  70%  of  the  time  to  plan  aircraft  re-routes;  to 
open,  close,  or  reconfigure  ATAs  and  DTAs;  and  to  anticipate  airport  closures.  Users  indicated 
that  the  TCWF  was  used  about  50%  of  the  time  or  less  to  anticipate  MIT  restrictions  and  in 
determining  holding  areas.  The  less  frequent  TCWF  use  for  anticipating  MIT  restrictions  may 
be  due  to  the  longer  range  planning  required  for  MIT  implementation  (more  than  the  60-minute 
forecast  can  provide).  Also,  based  on  ZME  interview  data,  some  users  indicated  the  TCWF  was 
beneficial  in  releasing  planes  from  holding.  However,  determining  holding  areas  was  not 
reported  and  may  be  attributable  to  the  less  frequent  use  of  the  product  for  this  task.  Finally,  the 
TCWF  was  used  less  than  10%  of  the  time  for  coordinating  military  release  areas.  This  result 
could  be  a  function  of  the  infrequency  in  which  this  task  is  performed. 
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TABLE  30.  FREQUENCY  SCORE  FOR  TCWF  USE  FOR  EACH  TASK  (ZME  ONLY) 


TASKS 

FREQUENCY 

Plan  Aircraft  Re-Routes 

4 

Anticipate  MIT  Restrictions 

3 

Open/Close/Reconfigure  ATAs 

4 

Open/Close/Reconfigure  DTAs 

4 

Determine  Holding  Areas 

2.5 

Anticipate  Airport  Closures 

4 

Coordinate  Military  Release  Areas 

1 

Aid  Situational  Awareness 

5 

Note:  1  =  Hardly  Ever  (less  than  10%),  2  =  Seldom  (about  30%),  3  =  Now  and  Then 
(about  50%),  4  =  Frequently  (about  70%),  5  =  Almost  Always  (at  least  90%),  and 
N/A  =  Product  not  used  for  task. 


Users  were  asked  if  ACCURACY  SCORES  effected  decisions  to  use  or  not  use  the  TCWF. 
If  affected,  then  criteria  used  for  using/not  using  the  product  was  solicited.  Table  31 
summarizes  accuracy  scoring  and  criteria  overall,  and  for  MEM  TRACON  and  ZME 
individually.  One  MEM  TRACON  user  reported  looking  at  accuracy  scores  to  determine 
product  use.  In  this  case,  criteria  for  not  using  the  product  were  scores  ranging  below  60%. 
Three  ZME  TMC  users  reported  using  accuracy  scores  to  determine  product  use.  Criteria 
identified  for  product  non-use  included  accuracy  ranges  below  65%  and  60%.  Three  ZME 
users  indicated  accuracy  scores  were  not  used. 


TABLE  31.  DECISION  TO  USE  PRODUCT  BASED  ON  ACCURACY  SCORES 


USER  GROUP 

YES 

CRITERIA  FOR 
NOT  USING 

NO 

N/A 

ALL  USERS  (N=12) 

4 

See  Below 

7 

1 

MEM  Sups  (N=5) 

1 

a.  Below  60% 

4 

0 

ZME  TMCs  (N=7) 

3 

a.  Below  65% 

b.  Below  60% 

3 

1 

6.3.4  Phase  2  Task  Questions. 


Table  32  summarizes  UTILITY  SCORES  for  identified  task  areas  for  MEM  TRACON  users. 
Utility  of  CWSU  information  was  not  measured  since  this  source  is  rarely  used.  MEM  users 
considered  the  TCWF  and  the  ITWS  SEP  component  equally  useful  for  all  task  areas.  Both 
ITWS  SEP  and  TCWF  were  ranked  "somewhat  useful"  for  most  task  areas.  For  "determining 
arrival  flow  rates"  and  "aiding  situational  awareness,"  both  products  were  considered  "extremely 
useful."  Coordinating  Severe  Weather  Avoidance  Programs  (SWAPs)  was  rated  "borderline,"  or 
"of  little  use." 


TABLE  32.  MEDIAN  UTILITY  SCORE  FOR  FORECAST  PRODUCTS 

(MEM  TRACON  ONLY) 


TASKS 

TCWF 

ITWS 

SEP 

Change  Runway  Configuration 

4 

4 

Determine  When  to  Close  Arrival  Gates 

4 

4 

Determine  When  to  Close  Departure  Gates 

4 

4 

Determine  When  to  Reopen  Departure  Gates 

4 

4 

Plan  Departure  Radials 

4 

4 

Determine  Departure  Flow  Rate 

4 

4 

Determine  Arrival  Flow  Rate 

5 

5 

Coordinate  SWAPs 

3 

3 

Aid  Situational  Awareness 

5 

5 

Note:  1  =  Of  No  Use,  2  =  Of  Little  Use,  3  =  Borderline,  4  =  Somewhat  Useful, 
5  =  Extremely  Useful,  and  N/A  =  Product  not  used  for  task. 


Table  33  summarizes  FREQUENCY  OF  USE  of  the  TCWF  to  complete  the  specific  MEM 
TRACON  tasks  during  convective  weather.  MEM  TRACON  users  utilized  the  TCWF  at  least 
70%  of  the  time  for  the  following  tasks: 

a.  Change  runway  configuration, 

b.  Determine  when  to  close  arrival  gates, 

c.  Determine  when  to  close  departure  gates, 

d.  Determine  when  to  reopen  departure  gates,  and 

e.  Plan  departure  radials. 

Users  also  indicated  that  the  TCWF  was  used  over  90%  of  the  time  to  aid  situational  awareness. 

For  three  task  areas,  the  frequency  of  TCWF  use  was  rated  50%  or  "now  and  then.".  These  tasks 
included: 

a.  Determine  departure  flow  rate, 

b.  Determine  arrival  flow  rate,  and 

c.  Coordinate  SWAPs 
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TABLE  33.  MEDIAN  FREQUENCY  SCORE  OF  TCWF  USE  PER  TASK 

(MEM  TRACON  ONLY) 


TASKS 

FREQUENCY 

Change  Runway  Configuration 

4 

Determine  When  to  Close  Arrival  Gates 

4 

Determine  When  to  Close  Departure  Gates 

4 

Determine  When  to  Reopen  Departure  Gates 

4 

Plan  Departure  Radials 

4 

Determine  Departure  Flow  Rate 

3 

Determine  Arrival  Flow  Rate 

3 

Coordinate  SWAPs 

3 

Aid  Situational  Awareness 

5 

Note:  1  =  Hardly  Ever  (less  than  10%),  2  =  Seldom  (about  30%),  3  =  Now  and  Then 
(about  50%),  4  =  Frequently  (about  70%),  5  =  Almost  Always  (at  least  90%),  and 
N/A  =  Product  not  used  for  task. 


Table  34  summarizes  the  UTILITY  SCORES  for  ZME  TMU  users.  ZME  users  ranked  the 
TCWF  as  "extremely  useful"  for  all  but  two  task  areas.  For  anticipating  MIT  and  coordinating 
military  release  areas,  the  TCWF  was  rated  "somewhat  useful".  ITWS  SEP  and  CWSU  were 
also  rated  as  "somewhat  useful"  for  these  tasks.  TCWF  usefulness  ratings  were  typically  higher 
than  usefulness  rating  scores  for  ITWS  SEP  and  CWSU.  ZME  TMU  users  also  rated  the  CWSU 
information  as  being  useful  for  most  tasks.  Both  TCWF  and  the  CWSU  were  considered 
extremely  useful  in  aiding  situational  awareness. 


TABLE  34.  MEDIAN  UTILITY  SCORE  FOR  FORECAST  PRODUCTS  (ZME  ONLY) 


TASKS 

TCWF 

ITWS  SEP 

CWSU 

Plan  Aircraft  Re-Routes 

5 

4 

4 

Anticipate  Miles-In-Trail  (MIT) 
Restrictions 

4 

4 

4 

Open/Close/Reconfigure  Arrival 
Transition  Areas  (ATAs) 

5 

4 

5 

Open/Close/Reconfigure  Departure 
Transition  Areas  (DTAs) 

5 

4 

4.5 

Determine  Holding  Areas 

5 

4 

4 

Anticipate  Airport  Closures 

5 

4 

4 

Coordinate  Military  Release  Areas 

4 

4 

4 

Aid  Situational  Awareness 

5 

4 

5 

Note:  1  =  Of  No  Use,  2  =  Of  Little  Use,  3  =  Borderline,  4  =  Somewhat  Useful,  5  =  Extremely 
Useful,  and  N/A  =  Product  not  used  for  task. 


Table  35  summarizes  FREQUENCY  OF  USE  of  the  TCWF  to  complete  the  specific  ZME  TMU 
tasks  during  convective  weather.  ZME  users  utilized  the  TCWF  most  frequently  for  anticipating 


airport  closures.  The  TCWF  was  used  at  least  70%  of  the  time  to  plan  aircraft  re-routes,  to  open, 
close,  or  reconfigure  ATAs  and  DTAs,  anticipate  MIT  restrictions,  determine  holding  areas,  and 
aid  situational  awareness.  Users  indicated  that  the  TCWF  was  used  about  50%  of  the  time  or 
less  to  coordinate  military  release  areas.  This  result  could  be  a  function  of  the  infrequency  in 
which  this  task  is  performed. 

TABLE  35.  FREQUENCY  SCORE  FOR  TCWF  USE  FOR  EACH  TASK  (ZME  ONLY) 


TASKS 

FREQUENCY 

Plan  Aircraft  Re-Routes 

4 

Anticipate  Miles-In-Trail  Restrictions 

4 

Open/Close/Reconfigure  ATAs 

4 

Open/Close/Reconfigure  DTAs 

4 

Determine  Holding  Areas 

4 

Anticipate  Airport  Closures 

5 

Coordinate  Military  Release  Areas 

3 

Aid  Situational  Awareness 

4 

Note:  1  =  Hardly  Ever  (less  than  10%),  2  =  Seldom  (about  30%),  3  =  Now  and  Then 
(about  50%),  4  =  Frequently  (about  70%),  5  =  Almost  Always  (at  least  90%),  and 
N/A  =  Product  not  used  for  task. 


6.4  WORKLOAD. 

As  described  in  section  4.2.3,  the  MCH  mental  workload  scale  was  used  to  measure  overall 
perceived  user  mental  workload  as  it  applied  to  operational  use  of  the  TCWF  display. 

Figures  9  and  10  are  bar  charts  of  pre-  and  post-median  ranking  scores  of  the  MCH  workload 
analysis  for  MEM  and  ZME  users.  The  10-point  workload  scale  rating  definitions  ranged  from: 

a.  1  -  very  easy,  highly  desirable.  Operator  mental  effort  is  minimal  and  desired 
performance  is  easily  attainable,  to 

b.  10  -  Impossible.  Instructed  task  cannot  be  accomplished  reliably. 

Lower  rating  scores  indicated  lower  perceived  workload  in  performance  of  any  of  the  identified 
tasks  during  a  convective  weather  event.  Conversely,  higher  scores  indicated  greater  perceived 
workload. 

Comparisons  between  MEM  TRACON  users’  MCH  scale  pre-and  post-workload  analysis 
showed  that  perceptions  of  operator  mental  workload  significantly  decreased  after  introduction 
of  the  TCWF  product  for  all  task  areas. 

For  ZME  users,  however,  workload  did  not  globally  decrease.  For  one  of  the  six  identified  ZME 
tasks,  anticipating  airport  closure,  perceived  operator  workload  actually  increased  with 
introduction  of  the  TCWF. 
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For  two  task  areas,  perceived  mental  workload  remained  the  same  (did  not  increase  or  decrease 
with  introduction  of  the  TCWF).  These  included: 

a.  Anticipate  MIT  restrictions,  and 

b.  Plan  departure  transition  areas. 

Post  score  comparisons  for  the  remaining  ZME  tasks  were  lower,  indicating  workload  had 
decreased  somewhat  after  TCWF  had  been  introduced.  These  tasks  included: 

a.  Plan  reroutes/deviations, 

b.  Plan  arrival  transition  areas,  and 

c.  Coordinate  military  release  areas. 


gates  gale  depcrturegate  depcrturegcrte 


BPre 
■  Post 


MEM  Tasks 


FIGURE  9.  PRE-AND  POST-MCH  WORKLOAD  COMPARISONS  -  MEM  TRACON 
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Plan  Aircraft  Re-  Anticipate  Miles-Irt-Trall  Change  AT  A  Change  DTA  Ant  icipale  Airport  CoonSnate  Military 

rout es/Devial ions  Restrictions  Closures  Release  Areas 


ZME  Tasks 


FIGURE  10.  PRE-AND  POST-MCH  WORKLOAD  COMPARISONS  -  ZME 
7.  CONCLUSIONS. 

Unless  noted,  conclusions  discussed  in  this  section  reflect  results  from  both  Phase  1  and  Phase  2 
of  the  TCWF  2000  Demonstration. 

a.  Overall  TCWF  Impression  -  Results  indicated  the  Terminal  Convective  Weather 
Forecast  (TCWF)  was  viewed  favorably  by  almost  all  participants  in  terms  of 
utility,  task  benefit,  benefit  beyond  Integrated  Terminal  Weather  System  (ITWS), 
and  interface  characteristics.  The  overall  impression  of  the  product  was  positive. 

b.  MEM  TRACON  Task  Benefit  -  All  Memphis  International  Airport  (MEM) 
Terminal  Radar  Approach  Control  (TRACON)  users  reported,  in  structured 
interview  responses,  that  TCWF  provided  benefit  in  performing  tasks  during 
operationally  significant  convective  events.  This  perception  was  also  reflected  in 
questionnaire  responses  where  utility  and  frequency  of  TCWF  use  was  measured. 
In  Phase  1,  TCWF  was  rated  "extremely  usefiil"  for  most  task  areas.  Results  for 
Phase  2  were  slightly  lower  with  most  tasks  receiving  "somewhat  useful"  ratings. 
No  task  areas  were  rated  "borderline"  or  negatively  (i.e.  no  use).  In  Phase  1, 
frequency  of  TCWF  use  in  all  task  areas  was  reported  to  be  70%  of  the  time  or 
more.  For  Phase  2,  however,  frequency  of  use  reportedly  decreased  to  "now  and 
then"  or  around  50%  of  the  time  in  three  task  areas: 

1.  Determine  Departure  Flow  Rate, 

2.  Determine  Arrival  Flow  Rate,  and 

3.  Coordinate  Severe  Weather  Avoidance  Programs  (SWAPs) 
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Why  MEM  TRACON  users  took  a  slightly  less  positive  view  of  the  TCWF  after  Phase  2 
completion  is  unknown. 

Recollections  from  MEM  TRACON  users  (n=3)  who  recalled  TCWF  performance  during 
convective  events  were  positive  overall.  Of  these,  TCWF  benefit  was  noted  for  determining 
when  to  come  in  or  out  of  holding  situations,  planning  runway  changes,  and  anticipating  gate 
impacts. 

c.  ZME  TMU  Task  Benefit  -  Memphis  ARTCC  (ZME)  Traffic  Management 
Coordinator  (TMCs')  perceptions  for  tasks  were  mostly  positive.  Although  found 
very  useful  for  MEM  airspace  impacts,  the  TCWF  was  not  adequate  for  all  airspace 
requirements,  which  comprises  a  majority  of  ZME  operations.  From  Phase  1, 
utility  scores  from  the  questionnaire  show  that  the  TCWF  was  found  "somewhat 
useful"  in  all  task  areas  with  the  exception  of  coordinating  military  release  areas. 
ITWS  and  the  Center  Weather  Service  Unit  (CWSU)  were  also  considered  "of  little 
use"  for  this  task.  As  noted  in  the  results  section  of  this  report,  coordinating 
military  release  areas  is  infrequently  performed.  From  Phase  2,  however,  utility 
scores  were  somewhat  higher.  With  the  exception  of  "Determine  MIT  restrictions  , 
and  "Coordinate  military  release  areas"  (rated  "somewhat  useful")  the  TCWF  was 
rated  "extremely  useful"  for  all  remaining  task  areas. 

TCWF  utility  was  rated  positively  for  most  ZME  tasks  in  Phase  1;  however,  the  product  was  not 
used  frequently  for  some  of  the  identified  task  areas.  These  tasks  included:  anticipating  miles- 
in-trail  (MIT)  restrictions,  determining  holding  areas,  and  coordinating  military  release  areas 
where  frequency  of  use  was  rated  at  50%  or  below. 

ZME  interview  responses  indicated  the  TCWF  was  useful  in  determining  when  to  release  aircraft 
from  holding;  however,  using  the  TCWF  to  determine  holding  restrictions  was  not  reported.  As 
reported  in  the  Results  section,  less  frequent  TCWF  use  for  anticipating  MIT  restrictions  may  be 
attributed  to  the  longer  strategic  planning  required  for  determining  MIT.  Again,  since  the 
"coordinating  military  release  areas"  task  is  seldom  performed,  this  may  explain  infrequency  of 
TCWF  use  for  this  activity. 

Phase  2  data  revealed  that  frequency  of  use  for  determining  holding  areas  and  anticipating  MIT 
restrictions  increased  to  "almost  always"  or  over  70%  of  the  time.  Although  in  Phase  1,  utility 
ratings  for  both  ITWS  Storm  Extrapolated  Position  (SEP)  and  the  TCWF  were  almost  identical, 
for  Phase  2,  TCWF  utility  scores  were  generally  higher  than  those  for  ITWS.  Users  reportedly 
preferred  the  TCWF  areal  storm  coverage  and  60-minute  animated  forecast  over  the  ITWS  SEP 
20-minute  leading  edge  line  depictions. 

Many  ZME  users  recalled  TCWF  performance  during  line-storm  convective  weather  events.  All 
recollections  were  positive,  with  TCWF  reportedly  aiding  in  decisions  involving  aircraft  holding, 
flow  control,  MEM  runway  and  gate  impacts,  as  well  as  enhancing  situational  awareness. 

d.  Situational  Awareness  -  Overall  perceptions  of  the  TCWF  benefiting  Situation 
Awareness  (SA)  were  positive.  Questionnaire  responses  from  both  ZME  and  MEM 
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TRACON  users  indicated  the  TCWF  was  extremely  useful  in  aiding  SA.  This  view 
was  also  reflected  in  interview  responses  where  both  ZME  and  MEM  TRACON 
users  noted  heightened  SA  as  one  of  the  TCWF  task  benefits. 

e.  TCWF  Interface  Suitability  -  Almost  all  TCWF  interface  components  (e.g.,  display 
window,  forecast  accuracy  scores)  were  viewed  favorably  overall  (both  ZME  and 
MEM  TRACON  users)  on  the  dimensions  of  utility,  readability,  and  ease  of  use. 
[All  components  were  rated  positively  (moderately  suitable  to  totally  suitable). 
Most  components  were  rated  totally  suitable.]  There  were  no  significant 
differences  between  median  ranking  scores  from  ZME  and  MEM  TRACON  users. 

f.  TCWF  Suggested  Improvements  -  Users  indicated  some  areas  for  TCWF 
improvement.  MEM  TRACON  users  suggested  the  following: 

1.  Keep  the  TCWF  separate  from  ITWS  -  See  item  i  (below). 

2.  Include  gust  front  and  wind  information  -  Including  this  information  would 
benefit  decision-making  regarding  changing  and  opening/closing  runways. 

3.  Integrate  with  ITWS  -  See  item  i  (below). 

4.  Improve  TCWF  training  -  Some  users  indicated  that  more  comprehensive 
training  might  have  lead  to  more  frequent  TCWF  use. 

5.  Weather  intensity  level  colors  should  be  the  same  as  those  used  on  ITWS.  The 
variations  of  the  color  yellow  to  imply  moderate  to  high  forecast  weather 
intensity  were  confusing  to  some  users.  Using  the  standard  color  intensities 
displayed  on  the  ITWS  Situation  Display  was  suggested. 


ZME  users  suggested: 

1.  Improve  growth  and  decay  capability  -  Show  where  cells  are  building  and 
when  and  where  they  will  dissipate.  Clearly  indicate  where  weather  will  be 
and  when. 

2.  Improve  forecast  accuracy  -  One  ZME  user  noted  that  the  TCWF  was  not 
relied  on  during  air-mass  type  convection.  Another  reported  little  confidence 
in  TCWF  accuracy  overall  and  did  not  use  the  product. 

3.  Integrate  with  ITWS.  See  item  i  (below). 

4.  Keep  TCWF  separate  from  ITWS.  See  item  i  (below). 

5.  Include  a  ZME  boundary  map  overlay  -  This  overlay  would  delineate  ZME 
airspace  boundaries  and  would  aid  in  locating  areas  of  convection  that  may 
affect  ZME  airspace. 
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6.  Increase  product  range  to  include  all  ZME  airspace  -  Forecast  weather 
requirements  for  ZME  go  beyond  the  MEM  Terminal  area.  Viewing  all  ZME 
airspace  may  contribute  to  planning  and  decision-making  for  enroute  as  well  as 
terminal  air  traffic. 

7.  Larger  display  monitor  -  A  larger  display  should  enhance  readability  of  TCWF 
graphics  and  text. 

8.  Extend  past  weather  depictions  -  Past  history  animations  longer  than  30 
minutes  would  enhance  weather  trending  perceptions. 

g.  Forecast  Requirements  -  Almost  all  MEM  TRACON  users  were  satisfied  with  the 
existing  30-  and  60-minute  predictions. 

Although  some  ZME  users  wanted  a  90-minute  forecast,  most  found  the  existing  30  and 
60-minute  forecasts  adequate.  In  order  to  effectively  control  all  air  traffic  within  ZME  airspace 
during  convective  weather  occurrences,  TMCs  require  planning  time  beyond  the  60-minute 
forecast  provided  by  the  TCWF.  However,  many  ZME  users  reported  only  using  the  TCWF  for 
MEM  airport  impacts.  Therefore,  the  TCWF  appears  to  be  more  of  a  tactical  tool  for  decisions 
regarding  MEM  arrivals  and  departures  and  less  of  a  strategic  tool  for  longer  range  enroute 
planning. 

h.  Range  Requirements  -  Given  the  available  ranges,  MEM  TRACON  users  preferred 
the  50  nautical  mile  (nm)  (TRACON)  range  presentation.  Consideration  of  a  range 
beyond  200  nm  was  unnecessary  since  50  nm  generally  encompassed  the  MEM 
area  of  interest. 

Most  ZME  users  viewed  the  30  and  50  nm  ranges.  However,  many  indicated  a  range  beyond 
200  nm  would  be  optimal.  Optimal  distances  ranged  from  400  to  600  nm.  Although  this 
preference  was  based  on  total  ZME  airspace  coverage,  results  showed  that  the  existing  200  nm 
range  was  seldom  used.  Therefore  it  is  questionable  if  longer-range  depictions  would  be  of  use 
to  the  typical  ZME  user. 

i.  Benefit  Beyond  ITWS  -  Most  users  reported  additional  benefit  to  having  TCWF  as 
well  as  ITWS  since  the  TCWF  provides  a  better  visual  presentation,  a  longer 
forecast,  and  better  indication  of  storm  movement.  Although  users  would  like  to 
have  the  TCWF  along  with  ITWS,  some  users  indicated  that  the  display  should  be 
separate  from  and  not  integrated  with  the  existing  ITWS  display.  Product 
integration  may  be  confusing  and  inhibit  TCWF/TTWS  accessibility.  Testing  of  an 
integrated  TCWF/TTWS  should  be  considered  before  implementation  of  the  two 
products  on  one  display. 

j.  Forecast  Accuracy  Scores  -  All  users  from  both  MEM  and  ZME,  who  responded  to 
the  question  of  whether  the  accuracy  scores  reflected  the  actual  accuracy  of  the 
TCWF,  indicated  that  the  scores  had  approximated  their  perception  of  the  product’s 
forecasting  performance. 
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Results  on  whether  the  accuracy  scores  affected  decisions  on  product  use  are  mixed.  In  Phase  1, 
almost  all  MEM  TRACON  users  reported  no  reliance  on  these  scores  for  basing  use/non-use 
decisions.  The  majority  of  ZME  users,  however,  did  factor  in  these  scores  before  using  TCWF 
information.  Although  currently  unknown,  this  result  could  reflect  individual  preferences  or 
perceptual  differences  between  Air  Route  Traffic  Control  Center  (ARTCC)  and  TRACON  users. 

Phase  2  results  showed  a  slight  increase  in  reliance  on  accuracy  scores  for  both  MEM  and  ZME 
users.  It  may  be  concluded,  that  with  greater  familiarity  and  confidence  in  accuracy  score 
depictions,  use  of  this  feature  may  have  increased. 

k.  Training  -  According  to  some  MEM  TRACON  Supervisors,  TCWF  training  should 
have  been  more  intensive.  With  better  training,  the  TCWF  may  have  been  more 
fully  utilized. 

Additionally,  as  discussed  in  the  "Factors  Affecting  Results"  section  of  this  report,  training 
material  should  not  have  contained  only  positive  (or  any)  results  from  past  TCWF 
Demonstrations.  User  bias  in  favor  of  a  product  or  system  may  have  confounded  demonstration 
results. 


1.  Workload  -  Modified  Cooper  Harper  (MCH)  workload  analysis  revealed  that  for 
MEM  TRACON  users,  perceived  operator  mental  workload  decreased  significantly 
in  all  identified  task  areas  after  introduction  of  the  TCWF. 

Workload  for  ZME  users,  however,  did  not  globally  decrease.  Workload  was  seen  to  increase 
for  the  task  area  of  anticipating  airport  closure.  For  two  other  tasks  areas:  1)  Anticipate  MIT 
restrictions,  and  2)  Plan  departure  transition  areas;  perceived  operator  mental  workload  remained 
the  same.  For  the  three  remaining  task  areas  measured,  workload  had  decreased  somewhat. 
Given  these  results,  it  is  difficult  to  generalize  the  affect  TCWF  had  on  workload.  It  may  be 
concluded  that  for  ZME  users,  the  TCWF  had  little  impact  (either  positively  or  negatively)  on 
perceived  mental  workload. 

8.  RECOMMENDATIONS. 

Based  on  conclusions  derived  from  this  report,  the  following  recommendations  are  proposed. 

a.  Integrated  TCWF/ITWS  Product  Testing  -  Due  to  the  absence  of  formal  test 
requirements,  ACB-630  cannot  make  recommendations  for  the  operational  use  of 
the  Terminal  Convective  Weather  Forecast  (TCWF).  However,  if  implementation 
of  the  TCWF  into  Integrated  Terminal  Weather  System  (ITWS)  proceeds  as 
proposed,  user  evaluations  should  be  conducted  prior  to  operational  availability  of 
this  new  system.  Even  if  ITWS  and  TCWF  individually  are  deemed  operationally 
suitable,  combining  the  two  creates  a  completely  new  system.  Feedback  from  users 
was  mixed  concerning  integration.  Users  expressed  concern  over  combining  the 
two  systems  on  one  platform.  These  concerns  included  but  were  not  limited  to: 
display  clutter,  individual  product  accessibility,  confusion  between  the  two  product 
displays,  and  adequate  display  window  space  for  a  viewable  product  presentation. 
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It  is  probable  that  other  issues  may  arise.  Evaluation/testing  should  facilitate 
identification  and  resolution  of  these  issues. 

b.  Training  -  Some  users  noted  the  TCWF  might  have  been  under-utilized  because 
training  was  insufficient.  It  is  recommended  that  in  future  demonstrations, 
subsequent  to  initial  product  training,  some  follow-up,  such  as  revisiting  the  site 
and  demonstrating  all  product  capabilities  during  an  operationally  significant 
weather  event,  be  initiated.  This  would  reinforce  previous  training  as  well  as 
demonstrate  optimal  product  utilization. 

It  is  also  recommended  that  training  material  focus  only  on  applications  and  use  of  the  product 
and  not  as  an  opportunity  for  product  promotion.  This  is  especially  critical  if  the  product  (in  this 
case,  TCWF)  is  to  undergo  an  unbiased  evaluation. 

c.  Algorithm  improvement  -  It  is  recommended  developers  continue  to  focus  research 
on  improving  the  TCWF  forecast  accuracy,  especially  for  air-mass  convective 
weather. 


9.  ACRONYMS. 

ARTCC 

Air  Route  Traffic  Control  Center 

ASD 

Aircraft  Situation  Display 

ASR 

Airport  Surveillance  Radar 

ATA 

Arrival  Transition  Area 

ATC 

Air  Traffic  Control 

ATCSCC 

Air  Traffic  Control  System  Command  Center 

ATIS 

Automatic  Information  Server 

AWRP 

Aviation  Weather  Research  Program 

CWSU 

Center  Weather  Service  Unit 

DFW 

Dallas/Ft.  Worth  International  Airport 

DTA 

Departure  Transition  Area 

GUI 

Graphical  User  Interface 

ITWS 

Integrated  Terminal  Weather  System 

km 

kilometer 

MCH 

Modified  Cooper  Harper 

MCO 

Orlando  International  Airport 

MEM 

Memphis  International  Airport 

MIT 

Miles-in-trail 

MIT/LL 

Massachusetts  Institute  of  Technology/Lincoln  Labs 

NEXRAD 

Next  Generation  Radar 

nm 

nautical  mile 

P3I 

Pre-planned  Product  Improvement 

SA 

Situational  Awareness 

SD 

Situation  Display 

SEP 

Storm  Extrapolated  Position 

Sup 

Supervisor 
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SWAP 

Severe  Weather  Avoidance  Program 

TCWF 

Terminal  Convective  Weather  Forecast 

TMC 

Traffic  Management  Coordinator 

TMU 

Traffic  Management  Unit 

TRACON 

Terminal  Radar  Approach  Control 

WARP 

Weather  and  Radar  Processor 

ZJX 

Jacksonville  ARTCC 

ZME 

Memphis  ARTCC 
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APPENDIX  A 


TRACON  AND  ARTCC  QUESTIONNAIRES 


2000  TERMINAL  CONVECTIVE 

WEATHER  FORECAST  DEMONSTRATION 

Phase  1 

MEMPHIS  TRACON  Supervisor/TMC 
QUESTIONNAIRE 


Prepared  by: 

Communication/Navigation/Surveillance 
Engineering  and  Test  Division,  Weather  Branch,  ACB-630 
William  J.  Hughes  Technical  Center 
Federal  Aviation  Administration 
Atlantic  City  International  Airport 
Atlantic  City,  NJ  08405 


A-l 


v  '  '  .  "  ■  '.  Instructions  -  •  '•*> 

The  purpose  of  this  questionnaire  is  to  obtain  feedback  from  users  regarding  the  Terminal 
Convective  Weather  Forecast. 

Feedback  from  users  is  a  very  important  component  of  the  Federal  Aviation  Administration 
(FAA)  William  J.  Hughes  Technical  Center  and  the  Massachusetts  Institute  of  Technology  Lincoln 
Laboratory  (MIT/LL)  demonstration  of  the  Terminal  Convective  Weather  Forecast.  Your  responses  to 
this  questionnaire  will  provide  important  information  for  use  in  future  iterations  of  the  Terminal 
Convective  Weather  Forecast.  Therefore,  please  respond  to  all  questions  as  honestly  and  thoroughly 
as  possible. 

Responses  to  this  questionnaire  will  remain  ANONYMOUS  and  CONFIDENTIAL.  A  report  will 
be  written  on  the  results  of  this  questionnaire,  summarizing  respondents’  comments;  however  no  one 
will  be  identified  or  associated  with  any  specific  comment.  Please  return  the  questionnaire  to  FAA 
Technical  Center  Evaluator. 


i:  i-U  «•  .V;  .  Definition  of  Terms  ■ 

Utility  -  This  refers  to  how  useful  the  TCWF  is  in  completing  job  tasks  related  to  traffic  planning 
as  impacted  by  convective  weather.  Elements  of  utility  would  include:  1)  effect  on  situational 
awareness;  2)  impacts  on  decision  making;  and/or  3)  impacts  on  workload. 

Ease  of  Use  -  This  refers  to  how  easy  the  TCWF  is  to  use  in  completing  job  tasks  (i.e. 
navigation,  button  use).  For  example,  a  user  may  consider  a  feature  easy  to  use  if  he/she  can 
discriminate  display  characters  and  symbols. 

Readability.-  This  refers  to  the  extent  the  information  displayed  is  easy  to  see,  readable,  meaningful, 
and  understandable.  For  example,  overlaid  text  may  be  too  small  to  read  or  obscured  by  other 
overlays. 

Situational  Awareness  -  Situational  Awareness,  or  SA,  has  been  defined  as  the  perception  of 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning, 
and  the  projection  of  their  status  in  the  near  future.  Essentially,  it  is  how  one  perceives  and 
understands  elements  in  the  present  situation  and  uses  this  information  to  anticipate  future  events. 


Terminal  Convective  Weather  Forecast  Information/Feature  Definitions  ;  0 

1 .  Display  Window  (Past/Forecast  Weather)  -  The  display  window  is  the  region  in  the  center  of  the 
screen  on  which  the  past  weather  and  forecast  weather  are  displayed.  The  window  can  display 
from  30  minutes  of  past  weather  to  60  minutes  of  forecasted  weather. 

2.  Forecast  Accuracy  Score  -  The  Forecast  Accuracy  Score  provides  real-time  scoring  information 
for  both  the  30  and  60-minute  forecasts. 

3.  Forecast  Weather  Color  Scale  -  The  forecast  weather  scale  shows  regions  in  which  the  growth 
and  decay  tracker  expects  a  moderate  to  high  probability  that  level  3  weather  (yellow  on  ITWS)  will 
occur  during  the  forecast  time  interval.  Solid  yellow  is  used  for  high  probability  regions  while  a 
hatched  yellow/gray  is  used  for  moderate  probability  regions. 
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r  <  Terminal  Convective  Weather  Forecast  Information/Feature  Definitions  (Cont’d)  ^ 

4.  Movie  Loop  -  The  movie  loop  selections  allow  the  user  to  select  a  variety  of  movie  loops  for 
displaying  data.  Loops  range  from  past  weather  only  to  30  or  60-minute  forecasts.  Past  weather 
and  forecasted  weather  can  be  looped  together  in  10-minute  increments.  Looping  can  occur  with  or 
without  past  weather. 

5.  Range  Indicator  -  The  Range  Indicator  is  located  to  the  left  of  the  display  window  just  below  the 
time  display.  The  range  indicator  shows  which  range  is  currently  selected  (i.e.,  50  NM  or  200  NM). 

6.  Relative  Time  Display  -  This  indicates  relative  time,  or  the  time  difference  from  the  current  time  to 
the  time  of  the  weather  shown  in  the  display  window.  This  time  will  be  positive  for  the  forecast 
images,  and  negative  for  past  weather  images.  The  font  will  be  white  when  the  images  are  looping 
and  will  turn  red  when  the  loop  has  been  stopped. 

7.  Start/Stop  Loop  Control  -  The  start/stop  loop  control  function  controls  the  starting  and  stopping  of 
forecast  loops.  It  also  enables  the  user  to  display  static  forecast  images. 


Y;  C  "  (ff"  Prbduct/Feature  Utility,  Ease  of  Use,  and  Readability 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  ease  of  use  and  readability 
of  the  Terminal  Convective  Weather  Forecast.  Please  refer  to  these  definitions  when 
responding.  Please  rate  the  product/feature  by  circling  the  appropriate  number. 


I 


Rating  Scale  Definitions 

5  -  Totally  Suitable.  This  response  indicates  the  product/feature  being  rated  is  totally 
appropriate  to  support  TRACON  Supervisors  in  the  performance  of  their  job  tasks. 

4  -  Moderately  Suitable.  This  response  indicates  the  product/feature  being  rated  is  suitable 
with  minimal  negative  impact  and  supports  TRACON  Supervisors  in  the  performance  of  their 
job  tasks. 

3  -  Suitable  with  Modification.  This  response  indicates  the  product/feature  being  rated  will  be 
appropriate  to  support  TRACON  Supervisors  in  the  performance  of  their  job  tasks;  however, 
modifications  would  improve  the  product. 

2  -  Moderately  Unsuitable.  This  response  indicates  the  product/feature  being  rated 
requires  some  significant  change  to  provide  appropriate  support  to  TRACON  Supervisors  in 
the  performance  of  their  job  tasks. 

1  -  Totally  Unsuitable.  This  response  indicates  the  product/feature  being  rated  will  have  a 
significant  negative  impact  to  TRACON  Supervisors  and  will  not  support  TMU  job  tasks. 

NA  -  You  have  never  used  the  product/feature  in  question. 
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Products/Features 

Totally  . 

Moderately 

Suitable  with 

Moderately 

-Totally 

n  k  ij.-f  w Suitable 

1 .  Display  Window  (Forecast  Weather) 

:  Suitable  ^ 

Modification 

Unsuitable 

Unsuitable 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

2.  Relative  Forecast  Time  Display 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

3.  Range  Indicators 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

4.  Forecast  Weather  Color  Scale 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

5.  Movie  Loop 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

6.  Start/Stop  Loop  Control 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

7.  Forecast  Accuracy  Score 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

8.  Overlays  (i.e.  jet  ways,  roads) 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

9.  Windowing  Options 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

A-4 


Products/Features 

10.  Status  Indicators 

a.  Utility 

b.  Readability 


Totally 

Suitable 


Moderately  Suitable  with  /  Moderately 
Suitable  Modification :  Unsuitable. 


Totally' t 
Unsuitable 


5  4  3  2  1  NA 

5  4  3  2  1  NA 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  how  useful  the  Terminal 
Convective  Weather  Forecast  (TCWF),  the  ITWS  Storm  Extrapolated  Position  (SEP),  and  CWSU 
Forecast  Information  were  in  performing  each  of  the  specified  tasks  during  convective  weather. 
Please  rate  the  aforementioned  products  INDEPENDENT  of  each  other.  For  example,  both  the 
TCWF  and  the  ITWS  SEP  can  receive  “Extremely  Useful”  ratings  for  a  specific  task  if 
appropriate.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Extremely  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  a  significant  positive  effect  on  decisions  related  to  the  specified  job  task. 

4  —  Somewhat  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  some  positive  effect  on  decisions  related  to  the  specific  job  task. 

3  -  Borderline.  This  response  indicates  the  convective  weather  forecast  information  had  little 
to  no  effect  on  decisions  related  to  the  specific  job  task. 

2  -  Of  Little  Use.  This  response  indicates  the  convective  weather  forecast  information  had 
some  negative  effect  on  decisions  related  to  the  specific  job  task. 

1  —  Of  No  Use.  This  response  indicates  the  convective  weather  forecast  information  had  a 
significant  negative  effect  on  decisions  related  to  the  specific  job  task. 

NA  -  You  have  never  used  the  product  in  question. 


Select  Job  Tasks  Extremely 

n'fiT'T?':  <f-C*WS»p  l  V. 

Somewhat 

.  r  w.;l- f 

Of  Little  - 

;iKOt  No  • 

Useful. 

Useful 

£  Borderline 

ff'Use55g;| 

SfcUseA 

1 1 .  Change  Runway  Configuration 

a.  TCWF 

5 

4 

3 

2 

i 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

i 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

i 

NA 

12.  Determine  When  to  Close  Arrival  Gate 

a.  TCWF 

5 

4 

3 

2 

i 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

i 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

i 

NA 
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Select  Job  Tasks  ':V  v  Extremely  rf  :s 

Somewhat 

Of  Little 

7  Of  No 

.  .  :  '  Useful 

13.  Determine  When  to  Close  Departure  Gate 

Useful 

Borderline 

Use  , ; > 

Use 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information  5 

14.  Determine  When  to  Reopen  Departure  Gate 

4 

3 

2 

1 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

15.  Plan  Departure  Radials 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

16.  Determine  Departure  Flow  Rate 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

17.  Determine  Arrival  Flow  Rate 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

18.  Coordinate  SWAPs 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

19.  Aid  Situational  Awareness 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

20.  Other 

a.  TCWF 

5 

4 

3 

2 

1 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 
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Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  frequency  (on  average)  in 
which  the  Terminal  Convective  Weather  Forecast  was  used  in  performing  the  specific  tasks 
listed.  Please  provide  frequency  ratings  for  TCWF  use  during  CONVECTIVE  WEATHER  EVENTS 
only.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Almost  Always.  This  response  indicates  the  TCWF  was  used  at  least  90%  of  the  time 
when  performing  the  specified  task  during  convective  weather  events. 

4  -  Frequently.  This  response  indicates  the  TCWF  was  used  about  70%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

3  -  Now  and  Then.  This  response  indicates  the  TCWF  was  used  about  50%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

2  -  Seldom.  This  response  indicates  the  TCWF  was  used  about  30%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

1  -  Hardly  Ever.  This  response  indicates  the  TCWF  was  used  less  than  10%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

NA  -  You  have  never  used  the  TCWF  operationally  during  convective  weather. 


21 .  Change  Runway  Configuration  5 

22.  Determine  When  to  Close  Arrival  Gate  5 

23.  Determine  When  to  Close  Departure  Gate  5 

24.  Determine  When  to  Reopen  Departure  Gate  5 


25.  Plan  Departure  Radials 

26.  Determine  Departure  Flow  Rate 

27.  Determine  Arrival  Flow  Rate 

28.  Coordinate  SWAPs 

29.  Aid  Situational  Awareness 

30.  Other _ 


5 

5 

5 

5 

5 

5 


4 

4 

4 

4 

4 

4 

4 

4 

4 

4 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


2 

2 

2 

2 

2 

2 

2 

2 

2 

2 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 


31 .  Given  the  types  of  decisions  that  you  make,  what  length  of  forecast  time  is  most  desired?  (Circle  one) 
30  min.  60  min.  90  min.  120  min.  Other _ min. 


Why?. 
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32.  What  percentage  of  time  did  you  view  weather  from  the  50  NM  range  versus  the  200  NM  range? 
50  NM  % _  200  NM  % _ 


33.  Would  a  range  beyond  200  nautical  miles  be  useful?  (Circle  one) 

Yes  No 

Specify  optimal  range.  Please  explain. 


34.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy?  (Circle 
one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

_ actual  forecast  was  better  than  the  forecast  accuracy  score 

_ actual  forecast  was  worse  than  the  forecast  accuracy  score 


35.  Did  the  30  and  60-minute  accuracy  scores  effect  your  decision  to  use  or  not  use  the  TCWF? 
(Circle  one) 


Yes  No 

If  yes,  how  did  it  effect  your  decision?  What  were  your  scoring  criteria  for  usinq/not  usinq  the 
TCWF  product? 


THANK  YOU  FOR  YOUR  TIME,  EFFORT,  AND  COOPERATION 

If  you  are  unable  to  return  this  questionnaire  to  an  on-site  FAA  Technical  Center  Evaluator, 
please  mail  the  completed  questionnaire  to  the  address  below. 

Cynthia  Fidalgo,  Raytheon  Technical  Services  Co. 

Weather  Branch,  ACB-630 
FAA  Technical  Center 
Atlantic  City  International  Airport,  NJ  08405 
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2000  TERMINAL  CONVECTIVE  WEATHER 
FORECAST  DEMONSTRATION 

Phase  1 

MEMPHIS  CENTER  TMU 
QUESTIONNAIRE 


Prepared  by: 

Communication/Navigation/Surveillance 
Engineering  and  Test  Division,  Weather  Branch,  ACB-630 
William  J.  Hughes  Technical  Center 
Federal  Aviation  Administration 
Atlantic  City  International  Airport 
Atlantic  City,  NJ  08405 
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Instructions 


The  purpose  of  this  questionnaire  is  to  obtain  feedback  from  users  regarding  the  Terminal 
Convective  Weather  Forecast. 


Feedback  from  users  is  a  very  important  component  of  the  Federal  Aviation  Administration 
(FAA)  William  J.  Hughes  Technical  Center  and  the  Massachusetts  Institute  of  Technology  Lincoln 
Laboratory  (MIT/LL)  demonstration  of  the  Terminal  Convective  Weather  Forecast.  Your  responses  to 
this  questionnaire  will  provide  important  information  for  use  in  future  iterations  of  the  Terminal 
Convective  Weather  Forecast.  Therefore,  please  respond  to  all  questions  as  honestly  and  thoroughly 
as  possible. 

Responses  to  this  questionnaire  will  remain  ANONYMOUS  and  CONFIDENTIAL.  A  report  will 
be  written  on  the  results  of  this  questionnaire,  summarizing  respondents’  comments;  however  no  one 
will  be  identified  or  associated  with  any  specific  comment.  Please  return  the  questionnaire  to  FAA 
Technical  Center  Evaluator. 


.3’. Definition  of  Terms 

Utility  -  This  refers  to  how  useful  the  TCWF  is  in  completing  job  tasks  related  to  traffic  planning 
as  impacted  by  convective  weather.  Elements  of  utility  would  include:  1)  effect  on  situational 
awareness;  2)  impacts  on  decision  making;  and/or  3)  impacts  on  workload. 

Ease  of  Use  -  This  refers  to  how  easy  the  TCWF  is  to  use  in  completing  job  tasks  (i.e. 
navigation,  button  use).  For  example,  a  user  may  consider  a  feature  easy  to  use  if  he/she  can 
discriminate  display  characters  and  symbols. 

Readability  -  This  refers  to  the  extent  the  information  displayed  is  easy  to  see,  readable,  meaningful, 
and  understandable.  For  example,  overlaid  text  may  be  too  small  to  read  or  obscured  by  other 
overlays. 

Situational  Awareness  -  Situational  Awareness,  or  SA,  has  been  defined  as  the  perception  of 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning, 
and  the  projection  of  their  status  in  the  near  future.  Essentially,  it  is  how  one  perceives  and 
understands  elements  in  the  present  situation  and  uses  this  information  to  anticipate  future  events. 


Terminal  Convective  Weather  Forecast  Information/Feature  Definitions  f  S  \  -  ;  i 

1.  Display  Window  (Past/Forecast  Weather)  -  The  display  window  is  the  region  in  the  center  of  the 
screen  on  which  the  past  weather  and  forecast  weather  are  displayed.  The  window  can  display 
from  30  minutes  of  past  weather  to  60  minutes  of  forecasted  weather. 

2.  Forecast  Accuracy  Score  -  The  Forecast  Accuracy  Score  provides  real-time  scoring  information 
for  both  the  30  and  60-minute  forecasts. 

3.  Forecast  Weather  Color  Scale  -  The  forecast  weather  scale  shows  regions  in  which  the  growth 
and  decay  tracker  expects  a  moderate  to  high  probability  that  level  3  weather  (yellow  on  ITWS)  will 
occur  during  the  forecast  time  interval.  Solid  yellow  is  used  for  high  probability  regions  while  a 
hatched  yellow/gray  is  used  for  moderate  probability  regions. 
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4.  Movie  Loop  -  The  movie  loop  selections  allow  the  user  to  select  a  variety  of  movie  loops  for 
displaying  data.  Loops  range  from  past  weather  only  to  30  or  60-minute  forecasts.  Past  weather 
and  forecasted  weather  can  be  looped  together  in  10-minute  increments.  Looping  can  occur  with  or 
without  past  weather. 

5.  Range  Indicator  -  The  Range  Indicator  is  located  to  the  left  of  the  display  window  just  below  the 
time  display.  The  range  indicator  shows  which  range  is  currently  selected  (i.e.,  50  NM  or  200  NM). 

6.  Relative  Time  Display  -  This  indicates  relative  time,  or  the  time  difference  from  the  current  time  to 
the  time  of  the  weather  shown  in  the  display  window.  This  time  will  be  positive  for  the  forecast 
images,  and  negative  for  past  weather  images.  The  font  will  be  white  when  the  images  are  looping 
and  will  turn  red  when  the  loop  has  been  stopped. 

7.  Start/Stop  Loop  Control  -  The  start/stop  loop  control  function  controls  the  starting  and  stopping  of 
forecast  loops.  It  also  enables  the  user  to  display  static  forecast  images. 


. .  Prod^  Use,  arid’ 

Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  ease  of  use  and  readability 
of  the  Terminal  Convective  Weather  Forecast.  Please  refer  to  these  definitions  when 


responding.  Please  rate  the  product/feature  by  circling  the  appropriate  number. 


Rating  Scale  Definitions 

5  -  Totally  Suitable.  This  response  indicates  the  product/feature  being  rated  is  totally 
appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the  performance  of  their  job 
tasks. 

4  -  Moderately  Suitable.  This  response  indicates  the  product/feature  being  rated  is  suitable 
with  minimal  negative  impact  and  supports  ARTCC  Traffic  Management  Coordinators  in  the 
performance  of  their  job  tasks. 

3  -  Suitable  with  Modification.  This  response  indicates  the  product/feature  being  rated  will  be 
appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the  performance  of  their  job 
tasks;  however,  modifications  would  improve  the  product. 

2  -  Moderately  Unsuitable.  This  response  indicates  the  product/feature  being  rated 
requires  some  significant  change  to  provide  appropriate  support  to  ARTCC  T raffic 
Management  Coordinators  in  the  performance  of  their  job  tasks. 

1  -  Totally  Unsuitable.  This  response  indicates  the  product/feature  being  rated  will  have  a 
significant  negative  impact  to  ARTCC  Traffic  Management  Coordinators  and  will  not  support 
TMU  job  tasks. 

NA  -  You  have  never  used  the  product/feature  in  question. 


A-ll 


Products/Features  '  ‘  '•  V*otaiiy^ 

M/. ;•.•  •i/'U.'v.  ,1  %  /  \:  v  i  fe;  :■ * '':A:Jrl: ;:u. .  Suitable 


1 .  Display  Window  (Forecast  Weather) 


a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

2.  Relative  Forecast  Time  Display 

a.  Utility  5 

b.  Readability  5 

3.  Range  Indicators 

a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

4.  Forecast  Weather  Color  Scale 

a.  Utility  5 

b.  Readability  5 

5.  Movie  Loop 

a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

6.  Start/Stop  Loop  Control 

a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

7.  Forecast  Accuracy  Score 

a.  Utility  5 

b.  Readability  5 

8.  Overlays  (i.e.  jetways,  roads) 

a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

9.  Windowing  Options 

a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 


Moderately  Suitable  with  Moderately  ^  :  ? 

Suitable ;  Modification  .Unsuitable  .(  Unsuitable 


4 

4 

4 


3 

3 

3 


2  1  NA 

2  1  NA 

2  1  NA 


4  3 

4  3 


2  1  NA 

2  1  NA 


4  3 

4  3 

4  3 


2  1  NA 

2  1  NA 

2  1  NA 


4  3 

4  3 


2  1  NA 

2  1  NA 


4  3 

4  3 

4  3 


2  1  NA 

2  1  NA 

2  1  NA 


4  3 

4  3 

4  3 


2  1  NA 

2  1  NA 

2  1  NA 


4  3 


2  1  NA 


4  3 


2  1  NA 


4  3 

4  3 

4  3 


2  1  NA 

2  1  NA 

2  1  NA 


4 

4 

4 


3 

3 

3 


2  1  NA 

2  1  NA 

2  1  NA 
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Products/Features 


Totally  Moderately  Suitable  with  Moderately  Totally 

Suitable  Suitable  *  Modification  Unsuitable  Unsuitable 


10.  Status  Indicators 

a.  Utility 

b.  Readability 


5  4  3  2  1  NA 

5  4  3  2  1  NA 


....  sf  V^-v  . 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  how  useful  the  Terminal 
Convective  Weather  Forecast  (TCWF),  the  ITWS  Storm  Extrapolated  Position  (SEP),  and  CWSU 
Forecast  Information  were  in  performing  each  of  the  specified  tasks  during  convective  weather. 
Please  rate  the  aforementioned  products  INDEPENDENT  of  each  other.  For  example,  both  the 
TCWF  and  the  ITWS  SEP  can  receive  “Extremely  Useful”  ratings  for  a  specific  task  if 
appropriate.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Extremely  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  a  significant  positive  effect  on  decisions  related  to  the  specified  job  task. 

4  -  Somewhat  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  some  positive  effect  on  decisions  related  to  the  specific  job  task. 

3  -  Borderline.  This  response  indicates  the  convective  weather  forecast  information  had  little 
to  no  effect  on  decisions  related  to  the  specific  job  task. 

2  -  Of  Little  Use.  This  response  indicates  the  convective  weather  forecast  information  had 
some  negative  effect  on  decisions  related  to  the  specific  job  task. 

1  -  Of  No  Use.  This  response  indicates  the  convective  weather  forecast  information  had  a 
significant  negative  effect  on  decisions  related  to  the  specific  job  task. 

NA  -  You  have  never  used  the  product  in  question. 


Select  Job  Tasks 


Extremely 

..Useful  . 


Somewhat 
.Useful  ? 


.  Of  Little 

Borderline^  •  Use 


Of  No 


1 1 .  Plan  Aircraft  Re-routes/Deviations 

a.  TCWF  5 

b.  ITWS  SEP  (Light  Blue  Contour)  5 

c.  CWSU  Forecast  Information  5 

12.  Anticipate  Miles-ln-Trail  Restrictions 

a.  TCWF  5 

b.  ITWS  SEP  (Light  Blue  Contour)  5 

c.  CWSU  Forecast  Information  5 


4  3  2  1  NA 

4  3  2  1  NA 

4  3  2  1  NA 


4  3  2  1  NA 

4  3  2  1  NA 

4  3  2  1  NA 
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Select  Job  Tasks 


Extremely  Somewhat 


Useful 


‘.Useful  ..  Borderline 


Of  Little 
Use 


13.  Open/Close/Reconfigure 
Arrival  Transition  Areas  (ATA) 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

14.  Open/Close/Reconfigure 
Departure  Transition  Areas  (DTA) 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

15.  Determine  Holding  Areas 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

16.  Anticipate  Airport  Closures 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

17.  Coordinate  Military  Release  Areas 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

18.  Aid  Situational  Awareness 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 

19.  Other _ _ 

a.  TCWF 

b.  ITWS  SEP  (Light  Blue  Contour) 

c.  CWSU  Forecast  Information 
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Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  frequency  (on  average)  in 
which  the  Terminal  Convective  Weather  Forecast  was  used  in  performing  the  specific  tasks 
listed.  Please  provide  frequency  ratings  for  TCWF  use  during  CONVECTIVE  WEATHER  EVENTS 
only.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Almost  Always.  This  response  indicates  the  TCWF  was  used  at  least  90%  of  the  time 
when  performing  the  specified  task  during  convective  weather  events. 


4  -  Frequently.  This  response  indicates  the  TCWF  was  used  about  70%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

3  -  Now  and  Then.  This  response  indicates  the  TCWF  was  used  about  50%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

2  -  Seldom.  This  response  indicates  the  TCWF  was  used  about  30%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


1  -  Hardly  Ever.  This  response  indicates  the  TCWF  was  used  less  than  10%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

NA  -  You  have  never  used  the  TCWF  operationally  during  convective  weather. 


^1 


20.  Plan  Aircraft  Re-routes/Deviations  5 

21.  Anticipate  Miles-ln-Trail  Restrictions  5 

22.  Open/Close/Reconfigure  ATAs  5 

23.  Open/Close/Reconfigure  DTAs  5 

24.  Determine  Holding  Areas  5 

25.  Anticipate  Airport  Closures  5 

26.  Coordinate  Military  Release  Areas  5 

27.  Aid  Situational  Awareness  5 

28.  Other _  5 


4 

4 

4 

4 

4 

4 

4 

4 

4 


3 

3 

3 

3 

3 

3 

3 

3 

3 


2 

2 

2 

2 

2 

2 

2 

2 

2 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 
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29.  Given  the  types  of  decisions  that  you  make,  what  length  of  forecast  time  is  most  desired*?  (circle 
one) 

30min-  60  min.  90  min.  120  min.  Other _ mi 

Why? _ 


30.  What  percentage  of  time  did  you  view  weather  from  the  50  NM  range  versus  the  200  NM  range? 

50  NM  % _  200  NM  % _ 

31 .  Would  a  range  beyond  200  nautical  miles  be  useful?  (circle  one) 

Yes  No 

Specify  optimal  range.  Please  explain. 


32.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy*?  (circle 
one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

- actual  forecast  was  better  than  the  forecast  accuracy  score 

- actual  forecast  was  worse  than  the  forecast  accuracy  score 


33.  Did  the  30  and  60-minute  accuracy  scores  effect  your  decision  to  use  or  not  use  the  TCWF*? 
(circle  one) 


Yes  No 

If  yes,  how  did  it  effect  your  decision?  What  were  your  scoring  criteria  for  usinq/not  usinq  the 
TCWF  product? 


THANK  YOU  FOR  YOUR  TIME,  EFFORT,  AND  COOPERATION 

If  you  are  unable  to  return  this  questionnaire  to  an  on-site  FAA  Technical  Center  Evaluator, 
please  mail  the  completed  questionnaire  to  the  address  below. 

Cynthia  Fidaigo,  Raytheon  Technical  Services  Co. 

Weather  Branch,  ACB-630 
FAA  Technical  Center 

Atlantic  City  International  Airport,  NJ  08405 
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2000  TERMINAL  CONVECTIVE  WEATHER 
FORECAST  DEMONSTRATION 

Phase  2 

MEMPHIS  TERMINAL  RADAR  CONTROL 
(TRACON)/TOWER  SUPERVISOR 
QUESTIONNAIRE 


Prepared  by: 

Communication/Navigation/Surveillance 
Engineering  and  Test  Division,  Weather  Branch,  ACB-630 
William  J.  Hughes  Technical  Center 
Federal  Aviation  Administration 
Atlantic  City  International  Airport 
Atlantic  City,  NJ  08405 
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:4T-ly-:v  •"  /.-i;:  5  fiv-'-v  ••  :v:  •  Instructions  '  r  7  v  'i/T’"?  - 

The  purpose  of  this  questionnaire  is  to  obtain  feedback  from  users  regarding  the  Terminal 
Convective  Weather  Forecast. 

Feedback  from  users  is  a  very  important  component  of  the  Federal  Aviation  Administration 
(FAA)  William  J.  Hughes  Technical  Center  and  the  Massachusetts  Institute  of  Technology  Lincoln 
Laboratory  (MIT/LL)  demonstration  of  the  Terminal  Convective  Weather  Forecast.  Your  responses  to 
this  questionnaire  will  provide  important  information  for  use  in  future  iterations  of  the  Terminal 

Convective  Weather  Forecast.  Therefore,  please  respond  to  all  questions  as  honestly  and  thoroughly 
as  possible.  a  3 

Responses  to  this  questionnaire  will  remain  ANONYMOUS  and  CONFIDENTIAL.  A  report  will 
be  written  on  the  results  of  this  questionnaire,  summarizing  respondents’  comments;  however  no  one 
will  be  identified  or  associated  with  any  specific  comment.  Please  return  the  questionnaire  to  FAA 
the  Technical  Center  Evaluator. 


l-'r Definition  of  Terms  !  -  .  .  . 

Utility  -  This  refers  to  how  useful  the  TCWF  is  in  completing  job  tasks  related  to  traffic  planning 
as  impacted  by  convective  weather.  Elements  of  utility  would  include:  1)  effect  on  situational 
awareness;  2)  impacts  on  decision  making;  and/or  3)  impacts  on  workload. 

Ease  of  Use  -  This  refers  to  how  easy  the  TCWF  is  to  use  in  completing  job  tasks  (i.e. 
navigation,  button  use).  For  example,  a  user  may  consider  a  feature  easy  to  use  if  he/she  can 
discriminate  display  characters  and  symbols. 

Readability  -  This  refers  to  the  extent  the  information  displayed  is  easy  to  see,  readable,  meaningful, 
and  understandable.  For  example,  overlaid  text  may  be  too  small  to  read  or  obscured  by  other 
overlays. 

Situational  Awareness  -  Situational  Awareness,  or  SA,  has  been  defined  as  the  perception  of 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning, 
and  the  projection  of  their  status  in  the  near  future.  Essentially,  it  is  how  one  perceives  and 
understands  elements  in  the  present  situation  and  uses  this  information  to  anticipate  future  events. 

-  Terminal  Convective  Weather  Forecast  Infbrmation/Feature  Definitions  ’jj  7 : 


1.  Display  Window  (Past/Forecast  Weather)  -  The  display  window  is  the  region  where  the  past 
weather  and  forecast  weather  are  displayed.  The  window  can  display  from  30  minutes  of  past 
weather  to  60  minutes  of  forecasted  weather. 

2.  Forecast  Accuracy  Score  -  The  Forecast  Accuracy  Score  provides  real-time  scoring  information 
for  both  the  30  and  60-minute  forecasts  at  TRACON  and  200  nm  ranges. 

3.  Forecast  Weather  Color  Scale  -  The  forecast  weather  scale  shows  regions  in  which  the  growth 
and  decay  tracker  expects  a  moderate  to  high  probability  that  level  3  weather  (yellow  on  ITWS)  will 
occur  during  the  forecast  time  interval.  Solid  yellow  is  used  for  high  probability  regions  while  a 
hatched  yellow/gray  is  used  for  moderate  probability  regions. 
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Terminal  Cbnv^tive  Weather  Forecast  Information/Feature  Defi  nitions  (Cont’d) 


4.  Movie  Loop  -  The  movie  loop  selections  allow  the  user  to  select  a  variety  of  movie  loops  for 
displaying  data.  Loops  range  from  past  weather  only  to  30  or  60-minute  forecasts.  Past  weather 
and  forecasted  weather  can  be  looped  together  in  10-minute  increments.  Looping  can  occur  with  or 
without  past  weather. 

5.  Relative  Time  Display  -  The  relative  time  display  is  located  on  the  bottom  left  of  the  display 
screen.  This  display  indicates  relative  time,  or  the  time  difference  from  the  current  time  to  the  time 
of  the  weather  shown  in  the  display  window.  This  time  will  be  positive  for  the  forecast  images,  and 
negative  for  past  weather  images.  The  font  will  be  white  when  the  images  are  looping  and  will  turn 
red  when  the  loop  has  been  stopped. 

6.  Start/Stop  Loop  Control  -  The  start/stop  loop  control  function  controls  the  starting  and  stopping  of 
forecast  loops.  It  also  enables  the  user  to  display  static  forecast  images. 

7.  Overlays  -  Overlays  are  accessed  via  the  overlay  menu  and  are  user  selected.  Overlays  consist 
of:  jetways,  victorways  range  rings,  TRACON,  outlying  airports,  counties,  MEM  sector,  and  states. 

8.  Product  Status  Buttons  -  Product  status  buttons  are  located  on  the  status  bar  on  the  upper  left 
side  of  the  display.  The  buttons  will  be  red  if  any  product  represented  by  the  button  is  not  available 
for  Memphis  airport  in  the  display  window.  Green  indicates  product  availability. 
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'.»&  ;  r ;  Y  /  .Product/Feature  Utility,  Ease  of  Use,  and  Readability  ^  '  ^ 

Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  ease  of  use  and  readability 
of  the  Terminal  Convective  Weather  Forecast.  Please  refer  to  these  definitions  when 
responding.  Please  rate  the  product/feature  by  circling  the  appropriate  number. 

Rating  Scale  Definitions 


5  -  Totally  Suitable.  This  response  indicates  the  product/feature  being  rated  is  totally 
appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the  performance  of 
their  job  tasks. 


4  -  Moderately  Suitable.  This  response  indicates  the  product/feature  being  rated  is 
suitable  with  minimal  negative  impact  and  supports  ARTCC  Traffic  Management 
Coordinators  in  the  performance  of  their  job  tasks. 

3  -  Suitable  with  Modification.  This  response  indicates  the  product/feature  being 
rated  is  appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the 
performance  of  their  job  tasks;  however,  modifications  would  improve  the  product. 

2  -  Moderately  Unsuitable.  This  response  indicates  the  product/feature  being  rated 
requires  some  significant  change  to  provide  appropriate  support  to  ARTCC  Traffic 
Management  Coordinators  in  the  performance  of  their  job  tasks. 

1 -Totally  Unsuitable.  This  response  indicates  the  product/feature  being  rated  has  a 
significant  negative  impact  to  ARTCC  Traffic  Management  Coordinators  and  will  not 
support  TMU  job  tasks. 


NA  -  You  have  never  used  the  product/feature  in  question. 


Moderately  Suitable  with  Moderately  %  Totally 
Suitable  ^Mwllfication^-  Unsuitable 


Unsuitable .  icf'.y 


1 .  Display  Window  (Forecast  Weather) 


a.  Utility  5 

b.  Readability  5 

c.  Ease  of  Use  5 

2.  Relative  Forecast  Time  Display 

a.  Utility  5 

b.  Readability  5 

3.  Forecast  Weather  Color  Scale 

a.  Utility  5 

b.  Readability  5 


4  3  2  1  NA 

4  3  2  1  NA 

4  3  2  1  NA 


4  3  2  1  NA 

4  3  2  1  NA 


4  3  2  1  NA 

4  3  2  1  NA 
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j.rrr>v.,v  *  yw^rs^tj-w  ya^flEW* 

Products/Features 

Totally  M 

oderate 

sly Suitable  with 

•.iT L  /i-V.-  iJMV}  ■yarjc-’pn'TH r<* J-,T  ‘  JJWiftTYB*'  • 

Moderately  ;  Totally 

«^S\alS*S«4MISSftas6» 

Suitable 

sg|||  Modification 

Unsuitable  ;  .  Unsuitable 

4.  Movie  Loop 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1  NA 

5.  Start/Stop  Loop  Control 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1  NA 

6.  Forecast  Accuracy  Score 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 

7.  Overlays  (i.e.  jetways,  roads) 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1  NA 

8.  Windowing  Options 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1  NA 

9.  Status  Indicators 

a.  Utility 

5 

4 

3 

2 

1  NA 

b.  Readability 

5 

4 

3 

2 

1  NA 
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i :  ,.V.  j V L. '  Utility  Per  Job  Task;  J;' W  W-  • 

Instructions:  The  five-point  scale  below  should  be  used  to  rate  how  useful  the  Terminal 
Convective  Weather  Forecast  (TCWF),  the  ITWS  Storm  Extrapolated  Position  (SEP),  and  CWSU 
Forecast  Information  were  in  performing  each  of  the  specified  tasks  during  convective  weather. 
Please  rate  the  aforementioned  products  INDEPENDENT  of  each  other.  For  example,  both  the 
TCWF  and  the  ITWS  SEP  can  receive  “Extremely  Useful”  ratings  for  a  specific  task  if 
appropriate.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Extremely  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  a  significant  positive  effect  on  decisions  related  to  the  specified  job  task. 

4  -  Somewhat  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  some  positive  effect  on  decisions  related  to  the  specific  job  task. 

3  -  Borderline.  This  response  indicates  the  convective  weather  forecast  information  had  little 
to  no  effect  on  decisions  related  to  the  specific  job  task. 


2  -  Of  Little  Use.  This  response  indicates  the  convective  weather  forecast  information  had 
some  negative  effect  on  decisions  related  to  the  specific  job  task. 

1  -  Of  No  Use.  This  response  indicates  the  convective  weather  forecast  information  had  a 
significant  negative  effect  on  decisions  related  to  the  specific  job  task. 


NA  -  You  have  never  used  the  product  in  question. 


Select  Job  Tasks' 


Extremely  :  Somewhat 
Useful  i  ;;,  Useful 


.  .  Of  Little 

Borderline  V  Use 


10.  Change  Runway  Configuration 

a.  TCWF  5  4 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4 

c.  CWSU  Forecast  Information  5  4 

1 1 .  Determine  When  to  Close  Arrival  Gate 

a.  TCWF  5  4 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4 

c.  CWSU  Forecast  Information  5  4 


12.  Determine  When  to  Close  Departure  Gate 


VofNo'"' 

Usd.”  •  „ 


1  NA 

1  NA 

1  NA 

1  NA 

1  NA 

1  NA 


a.  TCWF  5 

b.  ITWS  SEP  (Light  Blue  Contour)  5 

c.  CWSU  Forecast  Information  5 


4 

4 

4 


3  2  1  NA 

3  2  1  NA 

3  2  1  NA 
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Select  Job  Tasks 


Extrernely  Stjmewhat 
-  Useful Useful 


Of  Little  Of  No 
^Borderline  , Use  Use 


13.  Determine  When  to  Reopen  Departure  Gate 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

14.  Plan  Departure  Radials 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

15.  Determine  Departure  Flow  Rate 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

16.  Determine  Arrival  Flow  Rate 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

1 7.  Coordinate  SWAPs 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

18.  Aid  Situational  Awareness 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 

19.  Other _ 

a.  TCWF  5  4  3  2  1 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2  1 

c.  CWSU  Forecast  Information  5  4  3  2  1 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 
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Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  frequency  (on  average)  in 
which  the  Terminal  Convective  Weather  Forecast  was  used  in  performing  the  specific  tasks 
listed.  Please  provide  frequency  ratings  for  TCWF  use  during  CONVECTIVE  WEATHER  EVENTS 
only.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Almost  Always.  This  response  indicates  the  TCWF  was  used  at  least  90%  of  the  time 
when  performing  the  specified  task  during  convective  weather  events. 


4  -  Frequently.  This  response  indicates  the  TCWF  was  used  about  70%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


3  -  Now  and  Then.  This  response  indicates  the  TCWF  was  used  about  50%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


2  -  Seldom.  This  response  indicates  the  TCWF  was  used  about  30%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


1  -  Hardly  Ever.  This  response  indicates  the  TCWF  was  used  less  than  10%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

NA  -  You  have  never  used  the  TCWF  operationally  during  convective  weather. 


Frequenti 

y  Now 

Seldom  '“^4 

ftairdiv 

f-  -  '7!  v  — T5  - W/'""-  •  a 

waysifl 

kiand  Then' 

»ii.*T  .**»-&.  t-uayua 

■  Ever  ■ :  •-•-u  .Sf  -fKy 

20.  Change  Runway  Configuration 

5 

4 

3 

2 

1 

NA 

21 .  Determine  When  to  Close  Arrival  Gate 

5 

4 

3 

2 

1 

NA 

22.  Determine  When  to  Close  Departure  Gate  5 

4 

3 

2 

1 

NA 

23.  Determine  When  to  Reopen  Departure  Gate  5 

4 

3 

2 

1 

NA 

24.  Plan  Departure  Radials 

5 

4 

3 

2 

1 

NA 

25.  Determine  Departure  Flow  Rate 

5 

4 

3 

2 

1 

NA 

26.  Determine  Arrival  Flow  Rate 

5 

4 

3 

2 

1 

NA 

27.  Coordinate  SWAPs 

5 

4 

3 

2 

1 

NA 

28.  Aid  Situational  Awareness 

5 

4 

3 

2 

1 

NA 

29.  Other 

5 

4 

3 

2 

1 

NA 
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30.  Given  the  types  of  decisions  that  you  make,  what  length  of  forecast  time  is  most  desired?  (circle 
one) 

30  min.  60  min.  90  min.  120  min.  Other _ min. 

Why? _ _ _ 


31 .  What  nautical  mile  range(s)  do  you  use  most  often? 

5  mi _  15  mi _  30  mi _  TRACON__ _  100  mi _  200  mi 

32.  Would  a  range  beyond  200  nautical  miles  be  useful?  (circle  one) 

Yes  No 

Specify  optimal  range.  Please  explain. 


33.a.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy  for  the 
TRACON  range?  (circle  one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

_ actual  forecast  was  better  than  the  forecast  accuracy  score 

_ actual  forecast  was  worse  than  the  forecast  accuracy  score 


b.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy  for  the 
200  nm  range?  (circle  one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

_ actual  forecast  was  better  than  the  forecast  accuracy  score 

_ actual  forecast  was  worse  than  the  forecast  accuracy  score 


THANK  YOU  FOR  YOUR  TIME,  EFFORT,  AND  COOPERATION 

If  you  are  unable  to  return  this  questionnaire  to  an  on-site  FAA  Technical  Center  Evaluator, 
please  mail  the  completed  questionnaire  to  the  address  below. 

Cynthia  Fidalgo,  Raytheon  Technical  Services  Co. 

Weather  Branch,  ACB-630 
FAA  Technical  Center 

Atlantic  City  International  Airport,  NJ  08405 

A-25 


2000  TERMINAL  CONVECTIVE  WEATHER 
FORECAST  DEMONSTRATION 

Phase  2 

MEMPHIS  AIR  ROUTE  TRAFFIC  CONTROL  CENTER 

(ARTCC)  TMU 
QUESTIONNAIRE 


Prepared  by: 

Communication/Navigation/Surveillance 
Engineering  and  Test  Division,  Weather  Branch,  ACB-630 
William  J.  Hughes  Technical  Center 
Federal  Aviation  Administration 
Atlantic  City  International  Airport 
Atlantic  City,  NJ  08405 
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;;  Instructions  . 


The  purpose  of  this  questionnaire  is  to  obtain  feedback  from  users  regarding  the  Terminal 
Convective  Weather  Forecast. 

Feedback  from  users  is  a  very  important  component  of  the  Federal  Aviation  Administration 
(FAA)  William  J.  Hughes  Technical  Center  and  the  Massachusetts  Institute  of  Technology  Lincoln 
Laboratory  (MIT/LL)  demonstration  of  the  Terminal  Convective  Weather  Forecast.  Your  responses  to 
this  questionnaire  will  provide  important  information  for  use  in  future  iterations  of  the  Terminal 
Convective  Weather  Forecast.  Therefore,  please  respond  to  all  questions  as  honestly  and  thoroughly 
as  possible. 

Responses  to  this  questionnaire  will  remain  ANONYMOUS  and  CONFIDENTIAL.  A  report  will 
be  written  on  the  results  of  this  questionnaire,  summarizing  respondents’  comments;  however  no  one 
will  be  identified  or  associated  with  any  specific  comment.  Please  return  the  questionnaire  to  FAA 
the  Technical  Center  Evaluator. 


;  Definition  of  Terms  ;  '/  ,  , 


Utility  -  This  refers  to  how  useful  the  TCWF  is  in  completing  job  tasks  related  to  traffic  planning 
as  impacted  by  convective  weather.  Elements  of  utility  would  include:  1)  effect  on  situational 
awareness;  2)  impacts  on  decision  making;  and/or  3)  impacts  on  workload. 

Ease  of  Use  -  This  refers  to  how  easy  the  TCWF  is  to  use  in  completing  job  tasks  (i.e. 
navigation,  button  use).  For  example,  a  user  may  consider  a  feature  easy  to  use  if  he/she  can 
discriminate  display  characters  and  symbols. 

Readability  -  This  refers  to  the  extent  the  information  displayed  is  easy  to  see,  readable,  meaningful, 
and  understandable.  For  example,  overlaid  text  may  be  too  small  to  read  or  obscured  by  other 
overlays. 

Situational  Awareness  -  Situational  Awareness,  or  SA,  has  been  defined  as  the  perception  of 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning, 
and  the  projection  of  their  status  in  the  near  future.  Essentially,  it  is  how  one  perceives  and 
understands  elements  in  the  present  situation  and  uses  this  information  to  anticipate  future  events. 


W, 


Terminal  Convective  Weather  Forecast  Information/Feature  Definitions 


1 .  Display  Window  (Past/Forecast  Weather)  -  The  display  window  is  the  region  where  the  past 
weather  and  forecast  weather  are  displayed.  The  window  can  display  from  30  minutes  of  past 
weather  to  60  minutes  of  forecasted  weather. 

2.  Forecast  Accuracy  Score  -  The  Forecast  Accuracy  Score  provides  real-time  scoring  information 
for  both  the  30  and  60-minute  forecasts  at  TRACON  and  200  nm  ranges.. 

3.  Forecast  Weather  Color  Scale  -  The  forecast  weather  scale  shows  regions  in  which  the  growth 
and  decay  tracker  expects  a  moderate  to  high  probability  that  level  3  weather  (yellow  on  ITWS)  will 
occur  during  the  forecast  time  interval.  Solid  yellow  is  used  for  high  probability  regions  while  a 
hatched  yellow/gray  is  used  for  moderate  probability  regions. 
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Terminal  Convective  Weather  Forecast  Information/Feature  Definitions  (Cont’d) 

Movie  Loop  -  The  movie  loop  selections  allow  the  user  to  select  a  variety  of  movie  loops  for 
displaying  data.  Loops  range  from  past  weather  only  to  30  or  60-minute  forecasts.  Past  weather 
and  forecasted  weather  can  be  looped  together  in  1 0-minute  increments.  Looping  can  occur  with  or 
without  past  weather. 

5.  Relative  Time  Display  -  The  relative  time  display  is  located  on  the  bottom  left  of  the  display 
screen.  This  display  indicates  relative  time,  or  the  time  difference  from  the  current  time  to  the  time 
of  the  weather  shown  in  the  display  window.  This  time  will  be  positive  for  the  forecast  images,  and 
negative  for  past  weather  images.  The  font  will  be  white  when  the  images  are  looping  and  will  turn 
red  when  the  loop  has  been  stopped. 

6.  Start/Stop  Loop  Control  —  The  start/stop  loop  control  function  controls  the  starting  and  stopping  of 
forecast  loops.  It  also  enables  the  user  to  display  static  forecast  images. 

7.  Overlays  -  Overlays  are  accessed  via  the  overlay  menu  and  are  user  selected.  Overlays  consist 
of:  jetways,  victorways  range  rings,  TRACON,  outlying  airports,  counties,  MEM  sector,  and  states. 

8.  Product  Status  Buttons  -  Product  status  buttons  are  located  on  the  status  bar  on  the  upper  left 
side  of  the  display.  The  buttons  will  be  red  if  any  product  represented  by  the  button  is  not  available 
for  Memphis  airport  in  the  display  window.  Green  indicates  product  availability. 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  ease  of  use  and  readability 
of  the  Terminal  Convective  Weather  Forecast.  Please  refer  to  these  definitions  when 
responding.  Please  rate  the  product/feature  by  circling  the  appropriate  number. 


Rating  Scale  Definitions 

5  -  Totally  Suitable.  This  response  indicates  the  product/feature  being  rated  is  totally 
appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the  performance  of 
their  job  tasks. 

4  -  Moderately  Suitable.  This  response  indicates  the  product/feature  being  rated  is 
suitable  with  minimal  negative  impact  and  supports  ARTCC  Traffic  Management 
Coordinators  in  the  performance  of  their  job  tasks. 

3  -  Suitable  with  Modification.  This  response  indicates  the  product/feature  being 
rated  is  appropriate  to  support  ARTCC  Traffic  Management  Coordinators  in  the 
performance  of  their  job  tasks;  however,  modifications  would  improve  the  product. 

2  -  Moderately  Unsuitable.  This  response  indicates  the  product/feature  being  rated 
requires  some  significant  change  to  provide  appropriate  support  to  ARTCC  traffic 
Management  Coordinators  in  the  performance  of  their  job  tasks. 

1  -  Totally  Unsuitable.  This  response  indicates  the  product/feature  being  rated  has  a 
significant  negative  impact  to  ARTCC  Traffic  Management  Coordinators  and  will  not 
support  TMU  job  tasks. 


NA  -  You  have  never  used  the  product/feature  in  question. 


J»rpducts(l^aturestg^^pii|^ 

•Totally;’;^;? 
Suitable  A, 

Moderately 
S  Suitable  S 

^Sui&bl'^;Wiffi|o 

^■iModificatibnia 

Moderately i| 
Unsuitable 

§•;  Totally;-!/;^ 

1 Unsuitable; 

Stitts 

1 .  Display  Window  (Forecast  Weather) 

a.  Utility 

5 

4. 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

2.  Relative  Forecast  Time  Display 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

3.  Forecast  Weather  Color  Scale 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

Products/Features1  ?  " 

Totally 

Moderately 

Suitable  with 

Moderately ; 

'.Totally 

V  Suitable  /.; 

^ Suitable  ; 

Modification 

.  Unsuitable ; 

V  Unsuitable 

-  ft  -  : 

4.  Movie  Loop 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

5.  Start/Stop  Loop  Control 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

6.  Forecast  Accuracy  Score 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

7.  Overlays  (i.e.  jetways,  roads) 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

8.  Windowing  Options 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

c.  Ease  of  Use 

5 

4 

3 

2 

1 

NA 

9.  Product  Status  Buttons 

a.  Utility 

5 

4 

3 

2 

1 

NA 

b.  Readability 

5 

4 

3 

2 

1 

NA 

Per  Job  Task 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  how  useful  the  Terminal 
Convective  Weather  Forecast  (TCWF),  the  ITWS  Storm  Extrapolated  Position  (SEP),  and  CWSU 
Forecast  Information  were  in  performing  each  of  the  specified  tasks  during  convective  weather. 
Please  rate  the  aforementioned  products  INDEPENDENT  of  each  other.  For  example,  both  the 
TCWF  and  the  ITWS  SEP  can  receive  “Extremely  Useful”  ratings  for  a  specific  task  if 
appropriate.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Extremely  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  a  significant  positive  effect  on  decisions  related  to  the  specified  job  task. 

4  -  Somewhat  Useful.  This  response  indicates  the  convective  weather  forecast  information 
had  some  positive  effect  on  decisions  related  to  the  specific  job  task. 

3  -  Borderline.  This  response  indicates  the  convective  weather  forecast  information  had  little 
to  no  effect  on  decisions  related  to  the  specific  job  task. 

2  -  Of  Little  Use.  This  response  indicates  the  convective  weather  forecast  information  had 
some  negative  effect  on  decisions  related  to  the  specific  job  task. 

1  -  Of  No  Use.  This  response  indicates  the  convective  weather  forecast  information  had  a 
significant  negative  effect  on  decisions  related  to  the  specific  job  task. 


NA  -  You  have  never  used  the  product  in  question. 


Somewhat 
Useful: 


'Borderline 


Of  Little  ^ 

im 


Use; 


_ 


OffW 

Use 


10.  Plan  Aircraft  Re-routes/Deviations 

a.  TCWF  5  4  3 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3 

c.  CWSU  Forecast  Information  5  4  3 


2  1  NA 

2  1  NA 

2  1  NA 


11.  Anticipate  Miles-ln-Trail  Restrictions 

a.  TCWF  5  4  3  2 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3  2 

c.  CWSU  Forecast  Information  5  4  3  2 


1  NA 

1  NA 

1  NA 


12.  Open/Close/Reconfigure 
Arrival  Transition  Areas  (ATA) 

a.  TCWF  5  4  3 

b.  ITWS  SEP  (Light  Blue  Contour)  5  4  3 

c.  CWSU  Forecast  Information  5  4  3 


2  1  NA 

2  1  NA 

2  1  NA 
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Select  Job  Tasks  £  v.1  Extremely  . 

Somewhat 

.VW:v  ;  ‘  / 

Of  Little 

Cof  No 

'  -:r? 

:-aU.  >:.nl.Useful  ,; 

M  Useful 

.Borderline 

Use 

,Use?4; 

13.  Open/Close/Reconfigure 

Departure  Transition  Areas  (DTA) 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

14.  Determine  Holding  Areas 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

15.  Anticipate  Airport  Closures 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

16.  Coordinate  Military  Release  Areas 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

17.  Aid  Situational  Awareness 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 

18.  Other 

a.  TCWF 

5 

4 

3 

2 

1 

NA 

b.  ITWS  SEP  (Light  Blue  Contour) 

5 

4 

3 

2 

1 

NA 

c.  CWSU  Forecast  Information 

5 

4 

3 

2 

1 

NA 
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Frequency  of 


Instructions:  The  five-point  scale  below  should  be  used  to  rate  the  frequency  (on  average)  in 
which  the  Terminal  Convective  Weather  Forecast  was  used  in  performing  the  specific  tasks 
listed.  Please  provide  frequency  ratings  for  TCWF  use  during  CONVECTIVE  WEATHER  EVENTS 
only.  Please  refer  to  these  definitions  when  responding. 


Rating  Scale  Definitions 

5  -  Almost  Always.  This  response  indicates  the  TCWF  was  used  at  least  90%  of  the  time 
when  performing  the  specified  task  during  convective  weather  events. 

4  -  Frequently.  This  response  indicates  the  TCWF  was  used  about  70%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 

3  -  Now  and  Then.  This  response  indicates  the  TCWF  was  used  about  50%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


2  -  Seldom.  This  response  indicates  the  TCWF  was  used  about  30%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


1  -  Hardly  Ever.  This  response  indicates  the  TCWF  was  used  less  than  10%  of  the  time  when 
performing  the  specified  task  during  convective  weather  events. 


NA  -  You  have  never  used  the  TCWF  operationally  during  convective  weather. 


Sel  6Ct Jo  l> 

AlmBst 

Always] 

^'Frequently]  ^Novy  *; 
V.  :  and  Then 

|fei§§ggfj 

; - '  :;X,S  IV •' •; '  * •• ^  , “  .•/ 

Hardly] 

MMiMM 

jpl 

19.  Plan  Aircraft  Re-routes/Deviations 

5 

4 

3 

2 

i 

NA 

20.  Anticipate  Miles-ln-Trail  Restrictions 

5 

4 

3 

2 

i 

NA 

21 .  Open/Close/Reconfigure  AT As 

5 

4 

3 

2 

i 

NA 

22.  Open/Close/Reconfigure  DTAs 

5 

4 

3 

2 

i 

NA 

23.  Determine  Holding  Areas 

5 

4 

3 

2 

i 

NA 

24.  Anticipate  Airport  Closures 

5 

4 

3 

2 

i 

NA 

25.  Coordinate  Military  Release  Areas 

5 

4 

3 

2 

i 

NA 

26.  Aid  Situational  Awareness 

5 

4 

3 

2 

i 

NA 

27.  Other 

_  5 

4 

3 

2 

i 

NA 
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28.  Given  the  types  of  decisions  that  you  make,  what  length  of  forecast  time  is  most  desired?  (circle 
one) 

30  min.  60  min.  90  min.  120  min.  Other _ min. 


Why?. 


29.  What  nautical  mile  range(s)  do  you  use  most  often? 

5  mi _  15  mi _  30  mi _ _  TRACON _  100  mi _  200  mi. 

30.  Would  a  range  beyond  200  nautical  miles  be  useful?  (circle  one) 

Yes  No 

Specify  optimal  range.  Please  explain. 


31  .a.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy  for  the 
TRACON  range?  (circle  one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

_ actual  forecast  was  better  than  the  forecast  accuracy  score 

_ actual  forecast  was  worse  than  the  forecast  accuracy  score 


b.  Did  the  forecast  accuracy  score  match  your  impression  of  the  actual  forecast  accuracy  for  the 
200  nm  range?  (circle  one) 


Yes  No 

If  No,  which  of  the  following  is  true:  (mark  one) 

_ actual  forecast  was  better  than  the  forecast  accuracy  score 

_ actual  forecast  was  worse  than  the  forecast  accuracy  score 


THANK  YOU  FOR  YOUR  TIME,  EFFORT,  AND  COOPERATION 

If  you  are  unable  to  return  this  questionnaire  to  an  on-site  FAA  Technical  Center  Evaluator, 
please  mail  the  completed  questionnaire  to  the  address  below. 

Cynthia  Fidalgo,  Raytheon  Technical  Services  Co. 

Weather  Branch,  ACB-630 
FAA  Technical  Center 
Atlantic  City  International  Airport,  NJ  08405 
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APPENDIX  B 


STRUCTURED  INTERVIEW  RESPONSES 

TCWF  2000  Phase  1  Interview  Responses  -  Memphis 
A.  MEM  TRACON  Interview  Responses  -  Phase  1 
Question  1.  What  is  your  overall  impression  of  the  TCWF? 


User 

Response 

1 

Very  useful  when  weather  is  around  the  airport.  Find  it  accurate. 

2 

Very  useful. 

3 

Like  it.  Used  on  several  occasions.  Determines  where  weather  will  be 
for  runway  flow  patterns.  Useful. 

4 

It’s  awesome.  Love  it.  It’s  great. 

5 

Love  it.  Use  it  whenever  available.  Wish  it  were  in  the  Tower  Cab. 

2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  help? 


User 

Response 

1 

Helped  with  general  flow  (flow  control),  aircraft  holding,  and  to  anticipate 
runway  changes.  Prediction  of  weather  most  helpful. 

2 

Used  2  or  3  times.  Beneficial  for  holding  airplanes  and  coming  out  of 
holds  Forecasts  are  most  helpful. 

3 

Yes,  specifically  runway  flows,  holding,  and  terminal  routing,  i.e.,  tells  me 
where  weather  will  be  in  the  final  approach  corridor.  Also  gives  advance 
notice  of  when  weather  will  NOT  be  there. 

4 

Yes.  Used  to  plan  runway  configuration  and  traffic  management 
initiatives  such  as:  timing  AAR;  predicting  when  to  hold;  and  how  many 
aircraft  1  can  maneuver  at  a  time. 

5 

Used  for  planning  for  weather,  i.e.,  staffing.  Helps  determine  landing  and 
take-off  direction.  Use  the  10-minute  increments  of  weather  prediction. 
Can  do  precise  planning. 

3.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes, 
what  are  those  tasks  and  how  did  the  TCWF  hinder  them? 


User 

Response 

1 

No,  so  far.  Also,  there  hasn’t  been  a  lot  of  weather. 

2 

No. 

3 

No. 

4 

No. 

5 

No.  Usually  low  air-mass  scores,  so  won’t  rely  on  in  those  situations. 
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4.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial? 


User 

Response 

1 

Gives  better  idea  of  weather.  Helps  with  decisions  on  routing.  Smoothes 
out  the  operation.  Can  plan  rather  than  react.  Forecast  and  weather 
movement  most  useful. 

2 

Supplement  to  ITWS.  Forecast  and  movement  are  most  beneficial 

3 

Yes.  Is  a  good  prediction  tool.  Gives  picture  of  where  weather  will  be  in 
next  10,  20,  or  40  minutes. 

4 

Yes.  TCWF  shows  a  visual  prediction  vs.  prediction  lines  of  ITWS.  It’s 
easier  to  understand  a  picture.  The  forecast  and  forecast  counter  (counts 
down)  are  most  useful. 

5 

Good  benefit  with  ITWS.  ITWS  is  great  but  more  real  time.  The  forecast 
component  of  TCWF  is  a  good  supplement. 

5.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or 
out  of  Memphis  airport  during  convective  weather?  If  yes,  what  information  does 
the  TCWF  provide  that  helps? 


User 

Response 

1 

Yes,  see  above.  ^ 

2 

Yes.  Better  for  making  a  judgment  call  based  on  the  forecast.  Makes  job 
a  little  easier,  e.q.,  knowing  weather  will  be  across  final  in  20  minutes. 

3 

Yes.  See  above.  1 

4 

Yes.  See  above. 

5 

Yes.  Gives  me  more  information  to  make  decisions.  1  like  the  accuracy 
scores. 

6.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 


User 

Response 

1 

Predict  gust  fronts  and  wind  -  similar  to  ITWS.  Would  like  it  separate 
from  ITWS.  Easier  than  flipping  through  windows  on  1  display.  Prefer  2 
separate  monitors. 

2 

More  training  required.  Use  is  hit  or  miss.  The  class  was  limited. 

3 

No.  Don’t  think  it  would  be  good  to  integrate  with  the  ITWS.  Should  test 
in  an  integrated  mode  if  that  is  how  it  will  eventually  look.  Would  be 
useful  in  a  Tower  cab  for  runway  selection.  Would  help  in  co-decisions 
between  Tower  and  TRACON. 

4 

Would  like  to  see  TCWF  integrated  with  ITWS  on  a  separate  window. 

Add  wind  information  with  direction,  velocity  and  intensity  to  help  predict 
runway  shifts. 

Intensity  levels  are  confusing.  Use  levels  currently  used  on  ITWS  so  not 
as  much  of  an  adjustment  in  interpreting  them. 

Need  better  training  -  or  trainers  on  site  when  system  is  operational  and 
weather  breaks.  Would  be  qood  follow-up 

5 

No.  Like  the  stand-alone  display  -  two  monitors  side  by  side.  Do  not 
want  to  see  integrated  with  ITWS. 
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7.  Do  you  have  any  other  comments  or  concerns  regarding  the  forecast  product? 


User 

Response 

2 

Every  time  used,  the  TCWF  has  been  a  benefit. 

B.  ZME  TMU  Interview  Responses-  Phase  1 
1.  What  is  your  overall  impression  of  the  TCWF? 


User 

Response 

1 

Favorable.  Used  on  a  Fed  Ex  push  where  we  were  trying  to  get  all 
arrivals  in.  Were  able  to  get  all  but  six  in.  Effectively  used  to  reroute 
these  aircraft.  In  this  case,  the  product  was  accurate.  The  storm  was  a 
well-defined  line  from  the  southwest  up  the  river,  moving  to  the  North  and 
East  slowly.  This  is  an  ideal  situation  for  this  product.  Helped  to  manage 
the  flow. 

Haven’t  used  as  much  in  air-mass  conditions.  During  air-mass  situations 
we  are  more  reactionary.  Planning  is  more  dynamic  than  strategic. 

2 

Favorable.  Works  great.  Am  already  sold  on  ITWS.  Helped  last  night 
with  microburst  over  the  airport  and  the  airport  was  about  to  shut  down. 
Forecast  was  accurate.  FAA  should  buy  this. 

3 

Like  it.  Haven’t  used  much. 

4 

Good.  Is  good  with  ITWS.  Shows  where  weather  is  building.  However, 
haven’t  used  much. 

5 

Like  the  forecast  mode.  So  busy  there  has  been  no  opportunity  to  watch 
it.  East  and  West  coast  routes  force  traffic  on  top  of  us.  Spend  a  lot  of 
time  managing  traffic  volume. 

6 

Very  good  -  like  it.  Better  than  ITWS.  ITWS  doesn’t  show  motion  or 
history. 

7 

Doesn’t  do  anything  for  me.  1  don’t  base  anything  on  the  forecast. 
Accuracy  scores  are  low,  so  can’t  base  anything  on  that.  1  use  ITWS. 

8 

When  needed  1  used  it  -  mostly  for  MEM.  Would  use  more  if  1  had  more 
knowledge  about  it.  Need  to  get  more  familiar  with  it.  For  enroute 
weather,  1  use  the  TSD 

2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  help? 


User 

Response 

1 

Last  week,  MEM  was  closed  for  1  hr.  and  15  min.  due  to  air-mass 
conditions.  TCWF  was  not  used  to  forecast,  however  once  the  airport 
closed,  the  TCWF  was  used  to  see  when  to  take  planes  out  of  holding. 
TCWF  use  is  somewhat  marginal  but  sometimes  helps  with  planning. 

2 

Yes.  Which  routes  to  off-load.  Where  to  hold  airplanes.  Good  to 
establish  time  lines,  especially  for  fuel  critical  conditions. 

3 

N/A 

4 

Not  yet.  Can  see  where  it  will.  Am  a  little  reluctant  to  use  since  it’s 
experimental. 
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2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  help ?  (Cont’d) 


User 

Response 

5 

Sometimes.  When  weather  was  going  to  impact  a  gate  or  weather 
headed  to  a  southwest  arrival  fix  (Marvel).  Was  looking  at  the  storm 
prediction  and  movement  -  also  intensity  somewhat. 

6 

Really  a  terminal  product.  90%  of  use  is  for  MEM.  Great  to  determine: 
When  gates  will  shut  off, 

T  raffic  flows, 

Runway  configuration  problems, 

Decrease  in  AAR. 

Can  plan  accordingly.  Use  the  forecast  and  animated  display.  Easier  to 
visualize. 

7 

Never  used.  If  1  didn’t  have  ITWS,  1  would  look  at  it. 

8 

Yes.  Can  anticipate  gate  closings  and  move  planes  to  unaffected  gates. 
Timing  is  helpful.  Incremental  forecast  is  beneficial.  Used  it  along  with 
MEM  approach  -  we  were  both  looking  at  the  same  picture  and  can  focus 
on  the  same  view.  Projection  of  weather  is  good. 

3.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes, 
what  are  those  tasks  and  how  did  the  TCWF  hinder  them? 


User 

Response 

1 

No.  Would  never  base  a  decision  on  one  piece  of  information.  TCWF  is 
another  tool.  Gives  a  finer  picture  for  timing. 

2 

Never  “1 

3 

~No.  “  ~ 

4 

N/A 

5 

No. 

6 

Not  yet.  Only  use  if  weather  is  affectinq  MEM. 

7 

Saw  it  was  inaccurate.  Said  weather  would  be  impacting  the  field  but  it 
didn’t. 

8 

No.  Used  with  MEM  approach  -  made  a  check  and  balance. 

4.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial? 


User 

Response 

1 

Not  redundant  with  ITWS.  If  accuracy  percentages  are  good,  will  use. 
Used  for  timing  purposes,  i.e.,  when  will  airport  be  closed. 

2 

Has  benefit.  Needs  to  expand  beyond  the  terminal  area.  MEM  is  only 

20%  of  our  operations.  Need  a  better  prediction  of  where  weather  is 
going.  Although  the  30-minute  forecast  is  more  accurate,  need  the  same 
reliability  in  longer-term  projections. 
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4.  Has  the  TCWF  provided  benefit  beyond  ITWS?  if  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial?  (Cont’d) 


User 

Response 

3 

Yes,  some  benefit  beyond  ITWS.  Forecast  is  a  plus.  Also,  speed  of 
movement.  Allows  for  better  timing,  i.e.,  can  tell  airport  will  be  closed  in 

30  minutes.  Use  the  accuracy  scores  to  base  decisions  on. 

4 

Yes.  The  1-hour  forecast  and  to  see  if  weather  is  growing  or  decaying. 
Can  look  at  gates  to  see  where  the  weather  will  be  and  whether  or  when 
they  will  need  to  be  opened  or  closed. 

5 

Yes,  there  is  a  benefit  especially  with  the  prediction  mode. 

6 

See  #2 

7 

No  benefit  beyond  ITWS.  Forecast  is  the  only  benefit,  but  since  it’s 
inaccurate,  there  is  no  benefit. 

8 

Yes,  is  a  benefit,  but  only  for  the  airport. 

5.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or 
out  of  your  airspace  during  convective  weather?  if  yes,  what  information  does  the 
TCWF  provide  that  helps? 


User 

Response 

1 

Yes,  in  certain  situations  -  when  not  hectic.  Used  for  fine-tuning. 

2 

Yes.  Even  a  lot  of  enroute  traffic  flies  over  MEM.  30  to  60  minute 
forecast  is  the  most  useful  component. 

3 

Yes,  mostly  for  terminal  area.  200  nm  is  a  little  better  but  range  needs  to 
be  greater. 

4 

Yes,  helps  in  the  terminal  environment.  Enroute  is  questionable. 

5 

Good  for  planning  instead  of  waiting  for  approach  control  to  call.  Helped 
me  prepare  and  knew  what  to  do.  More  smooth  and  efficient. 

6 

Only  MEM  area.  1  have  a  problem  with  the  display  map.  There  is  no 

MEM  Center  boundary  map.  Not  a  good  quality  map.  In  the  long  range, 
overlays  are  ineffective. 

7 

N/A 

8 

Only  for  MEM.  Not  used  for  departures  but  for  arrivals  and  movement 
into  the  Terminal  area..  For  example,  last  night  1  knew  the  airport  would 
be  shut  off  in  10  minutes.  1  started  holding  planes  without  affecting  the 
traffic  flow.  Was  able  to  plan  instead  of  react. 

6.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 


User 

Response 

1 

Selectability  to  tighter  views.  Show  outlines  of  individual  cells  and  trace 
cells  better  over  certain  areas  of  the  airport.  Tell  when  area  will  be  clear 
or  where  weather  will  be,  e.g.,  thunderstorms  will  be  north  of  airport  for 
next  20  minutes  or  so. 

2 

Improve  accuracy  and  include  a  longer  range.  Integrate  with  ITWS 

3 

N/A 
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6.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 
(Cont’d) 


User 

Response 

4 

Presently,  don’t  know.  ~ 1 

5 

N/A 

6 

Bigger  monitor. 

7 

If  accuracy  improved  it  may  be  better.  Meant  more  for  a  terminal 
environment.  1  go  with  the  TRACON  decision  and  go  with  their  flow. 

Most  of  us  look  at  ITWS. 

8 

Don’t  know  yet.  When  1  become  more  familiar  with  it. 

7.  Do  you  have  any  other  comments  or  concerns  regarding  the  forecast  product? 
None  of  the  ZME  users  had  anything  to  add 
MEM  TRACON  Interview  Responses  -  Phase  2 

1.  What  is  your  overall  impression  of  the  product?  Has  that  impression  changed 
since  mid-July?  If  so,  how? 


User 

Response 

1 

Awesome,  great.  Makes  job  easier  since  don’t  have  to  guess  as  much  as  to 
where  convection  will  qo. 

2 

TCWF  is  an  asset  to  operations.  Appears  fairly  accurate.  A  better  initial  briefing 
on  the  product  would  have  helped  in  fully  utilizinq  and  understandinq  it 

3 

Have  not  used  much  lately  due  to  a  lack  of  weather. 

4 

It  is  really  good. 

5 

Good  product. 

2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  help  ? 


User 

Response 

1 

Has  helped  in  planning  purposes;  traffic  flow;  runway  configuration  (ITWS 
terminal  winds  are  helpful,  but  TCWF  helps  additionally  with  forecasts  of  wind 
shifts) 

2 

A  couple  of  times  the  product  helped  with  holding  at  fixes;  when  to  come  out  of 
holding;  which  fixes  would  open;  when  weather  would  leave  the  airport 

3 

The  60-minute  forecast  enables  planning  with  the  Center  if  holding  or  spacing  will 
be  required 

4 

Runway  changes;  flow  rates  -  what  to  expect  since  Center  determines;  arrivals 
and  departures;  being  able  to  predict  where  weather  will  be. 

5 

Determine  opening  and  closing  of  gates  for  arrivals  and  departures;  rerouting; 
can  give  Center  an  idea  of  when  can  open  up  arrival  gates;  is  a  good  planning 
tool. 
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3.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or 
out  of  your  airspace  during  convective  weather?  If  yes ,  what  information  does  the 
TCWF  provide  that  helps? 


User 

Response _ 

1 

Absolutely,  for  arrivals  and  departures.  TCWF  is  a  visual  aid.  While  forecast 
relative  time  is  useful,  the  actual  forecast  time  is  very  useful  in  determining  when 
storms  will  impact  airport  and  gates. 

2 

Yes,  see  responses  to  2  above. 

3 

Helps  due  to  planning. 

4 

Yes,  see  responses  to  2  above. 

5 

Yes,  would  think  so.  In  the  past  without  TCWF,  would  use  guess  and  experience 
to  estimate  how  fast  weather  is  moving. 

4.  Do  you  recall  any  convective  weather  scenarios  that  had  impacted  your 
airspace,  specifically  the  MEM  area 


User 

Response 

1 

Yesterday  a  line  passed  through.  Was  about  300-400  miles  long.  Was  able  to 
predict  when  aircraft  could  come  out  of  holding.  Nailed  forecast  with  only  1 
aircraft  having  to  divert.  Used  in  conjunction  with  ITWS,  which  gives  intensity 
levels.  TCWF  used  for  about  70-80%  of  planning.  Only  other  product  is  ASR-9 
weather,  which  is  really  not  used. 

2 

Used  3-4  times  with  line-storm  situations.  Used  to  determine  going  into  holdings 
and  coming  out,  and  runway  configurations.  Also  use  the  ASD.  In  addition, 
every  night  at  9  pm  a  conference  call  with  Federal  Express  weather  is  held. 

3 

No  situations  recalled. 

4 

Not  much  weather  lately.  In  early  summer  had  a  line  of  thunderstorms.  Used 
TCWF  in  the  ways  identified  in  2  above.  Also  used  ITWS;  ITWS  SEP  and  TCWF 
in  conjunction  with  one  another 

5 

No  recent  weather  that  can  recall. 

5.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes, 
what  are  those  tasks  and  how  did  the  TCWF  hinder  them? 


User 

Response 

1 

No. 

2 

No. 

3 

Is  a  hindrance  only  if  it  is  wrong. 

4 

No. 

5 

No. 

6.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial? 


User 

Response 

1 

TCWF  allows  seeing  the  prediction  rather  than  just  the  line  of  the  SEP. 

2 

The  forecast  enables  to  know  what  is  going  to  happen. 

6.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial?  (Cont’d) 


User 

Response 

3 

Forecast  since  ITWS  is  only  20  minutes;  TCWF  gives  longer  time  range.  It  gives 
a  better  idea  of  where  weather  will  be,  however,  ITWS  qives  an  idea  also 

4 

Cannot  think  of  any  other  than  the  extended  forecast  that  enables  being  able  to 
predict  where  weather  will  be. 

5 

ITWS  SEP  is  10-20  minutes  with  only  the  leading  edge;  thus,  doesn’t  give  entire 
area,  which  TCWF  does.  ASR-9  prone  to  false  weather  information. 

7.  Did  the  30  and  60  minute  forecast  accuracy  scores  effect  your  decision  to  use 
or  not  use  the  TCWF? 


User 

Response 

1 

No.  Although  one  time  when  TCWF  forecasts  were  not  occurring  as  predicted, 
the  accuracy  scores  reflected  this  with  low  scores.  Was  an  air-mass  situation. 
Actually  increased  confidence  in  the  product  since  it  showed  the  product  knew  it 
was  not  doing  well. 

2 

Accuracy  scores  were  high  when  TCWF  was  used.  Was  with  the  line-storms. 
Relied  upon  system  due  to  high  scores. 

3 

With  low  numbers  does  not  depend  on  the  product  as  much. 

4 

No.  Referred  to  them  and  took  with  a  grain  of  salt.  Go  ahead  and  make 
decisions  based  upon  product  image  and  can  update  in  30-40  minutes  if  was  not 
accurate.  TCWF  is  better  than  not  having  any  information. 

5 

Does  not  really  affect  decisions.  Tends  to  use  30  minute  forecast;  does  not  have 
much  use  for  60  minute  forecast  due  to  size  of  airspace. 

8.  Could  the  TCWF  be  modified  or  Improved  to  make  it  more  useful?  How? 


User 

Response 

1 

Would  like  to  have  intensity  levels.  ITWS  has  wind  direction  and  velocity,  which 
is  useful. 

2 

No.  However,  initial  briefing  was  lacking;  most  learning  was  on-the-job  pickinq 
things  up. 

3 

Make  available  to  controllers. 

4 

Make  more  user  friendly.  Not  familiar  enough  or  used  enough  to  know  how  to 
change  ranges  and  other  options  quickly.  If  does  change  something,  cannot 
figure  how  to  change  back.  Incorporating  into  ITWS  may  complicate  both  ITWS 
and  TCWF;  may  want  to  keep  separate  in  order  not  to  lose  usefulness. 

5 

Make  TCWF  more  accessible  to  controllers,  not  to  manipulate  but  to  have  display 
capability.  A  large  screen  capability  would  help.  Would  like  to  know  source  of 
weather  being  displayed,  that  is,  is  it  from  TDWR,  WSR-88D,  ASR-9.  Probably 
should  keep  ITWS  and  TCWF  as  two  separate  displays.  Never  having  seen 
integrated  with  ITWS,  would  not  be  sure  how  would  work.  TCWF  works  well 
displayed  as  is. 
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9.  Do  you  have  any  other  comments  or  concerns  regarding  the  forecast  product? 


User 

Response 

■ 

If  integrated  into  ITWS  would  need  a  larger  monitor.  The  current  scale  (size)  is 
nice,  would  not  like  to  have  a  smaller  version. 

TCWF  is  a  benefit  to  operations. 

3 

No. 

4 

Excellent  piece  of  equipment. 

5 

Need  rebriefing  on  product;  for  example  not  sure  how  to  change  and  use  looping 
capability.  Tend  to  forget  buttons  if  do  not  use  often.  Display  is  not  intuitive; 
more  use  to  drop-down  menus,  which  spell  out  options. 

ZME  TMU  Interview  Responses  -  Phase  2 

1.  What  is  your  overall  impression  of  the  product?  Has  that  impression  changed 
since  mid-July?  If  so,  how? 


User 

Response _ 

1 

Extremely  helpful 

2 

Has  no  use  for  TCWF.  It  is  an  approach  tool  used  for  runway  configurations  due 
to  wind  shifts.  MEM  airspace  is  large,  concentrating  on  one  area  does  not  help. 
Was  adamant  that  on  two  previous  occasions  had  told  evaluators  that  the 
product  was  not  useful. 

3 

Very  little  weather  in  the  last  4  months,  but  prior  to  that  was  pleased  with  the 
product.  Does  not  appear  to  be  as  accurate  with  sporadic  (air-mass) 
thunderstorms  as  with  lines.  More  confidence  with  the  forecast  of  lines. 

4 

A  qood  tool,  useful. 

5 

Likes  the  product.  Looks  at  it  more  than  at  ITWS. 

6 

Like  a  lot  better  than  ITWS,  has  forecast  which  appears  accurate. 

7 

Likes  the  product. 

8 

! 

Has  been  pleased;  there  has  been  a  number  of  occasions  where  the  product  has 
been  helpful. 

9 

Likes  it. 

10 

Impressed. 

11 

Very  qood. 

12 

Have  not  used  TCWF  much,  only  been  in  unit  about  6  months;  what  has  seen 
appears  to  be  good. 
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2.  Has  the  product  helped  you  perform  any  specific  tasks?  If  yes,  what  are  those 
tasks  and  how  did  the  TCWF  help  ? 


User 

Response 

1 

Assists  coordination  with  terminal  facility  (TRACON)  through  shared  situational 
awareness;  flow  in  and  out  of  MEM;  helps  some  with  flow  in  and  out  of  DFW  and 
ATL;  use  for  enroute  of  timing  of  aircraft  if  operating  in  MEM  area;  deciding 
whether  to  use  miles-in-trail  and  holds. 

2 

"n/a  "  ~  - - - - 

3 

Anticipation  of  impact  of  weather  on  arrival  and  departure  gates;  can  pre-plan 
movement  of  aircraft 

4 

Estimate  when  weather  will  impact  the  airport  at  MEM;  for  holding  aircraft; 
shutting  off  gates.  TCWF  is  easier  and  more  precise  than  MWP;  shows  weather 
in  greater  detail 

5 

Look  at  arrivals  to  see  if  need  to  plan;  use  TCWF  forecasts  to  see  if  need  to  hold 
or  not. 

6 

Plan  off  loads  going  to  MEM;  mainly  with  MEM  inbound  push;  if  TCWF  shows 
convective  weather  over  a  gate  in  30  minutes,  than  can  adjust. 

7 

Yes,  when  a  line  of  weather  occurs  at  the  same  time  as  arrival  bank  at  airport, 
then  allows  to  be  prepared  to  go  into  holding  patterns.  Aircraft  from  the  east  and 
northeast  are  impacted  the  most;  can  determine  which  ones  can  get  in  ahead  of 
an  approaching  line  and  which  ones  will  need  to  be  rerouted;  aircraft  from  the 
west  tend  to  be  trailing  the  weather  so  impact  is  not  as  great 

8 

When  airport  will  clear;  what  fixes  will  be  impacted  and  when;  helps  in  choosing 
when  to  move  aircraft  so  can  get  into  airport  as  quickly  as  possible  when  weather 
clears.  Jet  aircraft  have  many  options,  with  TCWF  can  plan. 

9 

Forecasting  when  airport  will  go  down;  gate  balancing;  weather  avoidance  for 
arrivals 

10 

l  Yes,  used  for  in  and  around  MEM  rather  than  whole  job  responsibilities 

11 

Routes  to  MEM;  miles-in-trail  plans;  talk  to  approach  control;  may  affect 
departure  routes. 

12 

No  decisions,  but  have  not  had  the  opportunity. 

3.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or 
out  of  your  airspace  during  convective  weather?  If  yes,  what  information  does  the 
TCWF  provide  that  helps? 


User 

Response 

1 

Yes,  see  2  above.  i 

2 

"N/A  ~  - - - 

3 

I  uwf  is  used  as  a  tool  with  lines;  takes  the  information  qiven  as  is  and  uses  it 

4 

Forecast  movement  is  beneficial;  presentation  of  gust  fronts  is  helpful 

5 

Yes,  based  upon  TCWF  forecast. 

6 

Yes,  can  pre-plan  rather  than  react. 

7 

Yes,  see  2  above;  is  a  specific  use,  but  an  important  one  especially  for  hub 
airports. 

8 

Yes,  see  2  above. 

9 

Yes,  forecast  allows  to  be  proactive  in  bringing  traffic  on  backside  1 
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3.  Has  the  TCWF  enabled  you  to  more  efficiently  handle  the  flow  of  traffic  into  or 
out  of  your  airspace  during  convective  weather?  If  yes,  what  information  does  the 
TCWF  provide  that  helps?  (Cont’d) 


User 

Response 

10 

Yes,  determining  gates  that  may  be  closed. 

11 

Yes,  see  2  above. 

12 

N/A 

4.  Do  you  recall  any  convective  weather  scenarios  that  had  impacted  your 
airspace,  specifically  the  MEM  area. 


User 

Response 

1 

Last  night  had  a  line  of  thunderstorms  from  about  100  miles  south  of  MEM  to 
about  80  miles  north  of  MEM.  Used  to  determine  miles-in-trail  and  holding;  were 
able  to  spread  aircraft  out  to  make  a  manageable  situation;  held  aircraft  and 
brought  them  in  on  the  back  side  of  the  line;  was  able  to  minimize  holding  time 
and  bring  aircraft  in  as  soon  as  weather  cleared  MEM.  Product  had  good 
accuracy  in  this  event. 

2 

Uses  aircraft  reports  and  lightning  strike  information.  Does  not  rely  on  forecasts. 

3 

Yesterday’s  weather  situation  was  a  good  example  of  how  TCWF  has  been  used. 
Was  able  to  predict  approximate  times  to  move  aircraft  to  gates;  did  not  have  to 
divert  because  of  TCWF  information. 

4 

N/A 

5 

Line  of  storms  on  1 1/6  were  forecasted  accurately  by  TCWF;  was  only  off  by  10 
minutes  with  airport  arrival.  Due  to  TCWF  information,  knew  had  to  move  aircraft 
north  around  line  and  bring  aircraft  in  on  backside  of  line;  did  not  hold  aircraft  out 
in  front  of  line.  Relied  solely  upon  the  TCWF. 

6 

Predominantly  light  rain  2  days  ago;  did  not  have  an  impact  so  did  not  use 
product.  Cannot  recall  any  specific  events. 

7 

See  2  above;  describes  exactly  what  happened.  TCWF  is  very  effective  with 
line-storms. 

8 

Use  ITWS,  however,  TCWF  forecasts  are  easier;  can  tell  if  airport  will  be 
impacted  and  if  need  to  use  miles-in-trail  and  spacing.  With  ITWS,  have  to 
assume  airport  will  be  impacted  due  to  the  lack  of  forecast  information.  Line- 
storms  tend  to  be  more  accurate;  isolated  storms  are  harder  to  predict,  but  if 
TCWF  is  forecasting  something  with  isolated  storms,  then  know  to  be  aware. 

9 

Yesterday  had  a  line;  used  as  described  in  2  above.  CWSU  provides  information 
along  with  WARP.  Will  discuss  TCWF  with  CWSU  to  see  what  they  think.  Refer 
to  WARP  to  get  bigger  picture. 

10 

Did  not  use  yesterday  during  weather  event;  weather  was  not  intense  enough, 
mostly  just  precipitation. 

11 

Any  time  there  has  been  convective  weather,  has  used  TCWF  to  monitor  the 
weather;  passes  information  to  other  Centers  to  avoid  overloading  sectors;  has 
used  for  justification  to  ATCSCC. 

12 

N/A 
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5.  Has  the  product  hindered  your  ability  to  perform  any  specific  tasks?  If  yes, 
what  are  those  tasks  and  how  did  the  TCWF  hinder  them  ? 


User 

Response 

1 

No.  Users  realized  that  no  product  is  100%.  1 

2 

~No  - - 

3 

~No  '  - - 

4 

"No  ~  - 

5 

“No  —  - - 

6 

No  - 

7 

~No  “  - - 

8 

"No  —  “  - - 

9 

"No  "  - - — 

10 

"No  - - 

11 

"No  ~  - - 

12 

“No  ~  - - 

# 

( 


6.  Has  the  TCWF  provided  benefit  beyond  ITWS?  If  yes,  what  features  on  the 
TCWF  have  been  more  helpful?  If  no,  why  is  ITWS  more  beneficial? 


User 

Response 

1 

Forecast  product,  but  still  use  in  conjunction  with  ITWS. 

2 

~Wa  ~  - - 

3 

No.  ITWS  gives  winds,  if  TCWF  included  wind  information  that  would  be  helpful. 
Presentation  is  more  helpful  than  ITWS. 

4 

Does  not  always  accept  TCWF  forecasts;  waits  and  sees  what  the  product  is 
doing.  Uses  information  as  a  possibility  and  consider  what  to  do  if  forecast 
happens.  TCWF  does  not  add  a  lot  over  ITWS. 

5 

Gives  a  forecast  that  ITWS  does  not  have;  ITWS  gives  current  weather,  TCWF 
gives  what  will  happen. 

6 

Forecast  is  only  area 

7 

ITWS  shows  current  weather,  TCWF  forecast  is  an  enhancement. 

8 

Looping  feature  is  helpful. 

9 

TCWF  has  enhanced  features;  forecast  mode  is  the  best;  overall  layout  seems 
more  user  friendly;  changing  ranges  is  fairly  easy;  looping  is  beneficial. 

10 

Forecast  is  an  advantage  over  ITWS;  animation. 

11 

Longer  forecast;  uses  ITWS  and  TCWF  in  a  similar  manner;  CWSU  turns  TCWF 
on. 

12 

Forecast  is  better;  very  accurate. 

t 
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7.  Did  the  30-  and  60-minute  forecast  accuracy  scores  effect  your  decision  to  use 
or  not  use  the  TCWF? 


User 

Response 

1 

If  low  reliability,  still  use  product  but  give  less  weight  to  forecast.  If  accuracy  > 
40%,  then  give  more  confidence.  Acceptance  of  the  reliability  has  improved. 

2 

Cannot  base  decisions  on  40-50%  forecast  accuracy.  Accuracy  was  not  "there" 
so  no  confidence  in  product. 

3 

Does  not  pay  attention  to  scores.  If  product  is  turned  on,  then  it  is  for  a  reason 
and  will  trust  the  product. 

4 

No,  does  not  watch  forecast  accuracy  much. 

5 

Lack  of  scores  affected  confidence;  was  not  there  initially  when  thought  should 
have  been.  If  scores  of  70  and  higher,  then  have  confidence  in  product. 

6 

No;  product  is  accurate  with  lines;  if  not  a  line,  then  is  not  as  accurate.  Seems  to 
do  a  better  job  with  level  4  and  above. _ 

7 

Yes;  lines  are  more  accurate  than  air-mass.  This  taken  into  account  especially  if 
numbers  are  down  and  at  longer  forecast  times  (>  40  minutes).  With  shorter 
forecast  periods,  tend  to  look  more  at  how  storms  are  tracking  via  the  looping 
capability;  looping  is  useful  even  for  air-mass;  user  has  to  do  some  figuring  for 
self  by  looking  at  trends  in  growth  and  movement. 

8 

1 

Try  to  pay  attention  to  scores;  if  not  as  accurate  then  do  not  pay  attention  to 
scores  as  much;  hard  to  evaluate  where  a  cut-off  may  be;  when  using  TCWF,  do 
not  have  time  to  look  at  scores;  is  using  about  75%  as  a  criteria,  but  have  to  use 
product  over  time  to  get  comfortable  with  it. 

9 

Slightly;  when  weather  exists,  things  are  happening  fast,  may  only  glance  at 
scores;  if  product  shows  something  different  than  other  sources  then  may  look  at 
scores. 

10 

Does  not  pay  much  attention  to  scores;  is  a  waste  of  space. 

11 

Look  mainly  at  the  forecast  display  to  see  how  well  storms  are  being  predicted; 
use  scores  less  than  50%  of  the  time. 

12 

No,  not  familiar  with  scores. 

8.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 


User 

Response 

1 

Prefers  TCWF  and  ITWS  on  same  monitor  displayed  side-by-side;  likes  size  of 
current  TCWF.  ITWS  runway  winds  use  to  be  always  available,  now  have  to 
specify;  would  like  to  have  wind  information  readily  available. 

2 

N/A 

3 

Nothing  aware  of. 

4 

Would  like  to  have  at  greater  ranges.  If  could  do  mosaics  of  convective  weather, 
that  would  be  useful.  Would  like  to  have  for  more  facilities. 

5 

No;  have  not  used  much  due  to  lack  of  weather. 

6 

Move  down  from  top  shelf  in  TMU  so  can  see  it  better. 

7 

No,  happy  with  it. 

8 

Feels  there  is  an  accuracy  issue;  at  what  point  is  the  product  reliable;  does  not 
know  what  figures  mean.  Would  like  fixes  on  display  so  can  know  exactly  where 
weather  is.  Likes  having  a  separate  display. 
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8.  Could  the  TCWF  be  modified  or  improved  to  make  it  more  useful?  How? 
(Cont’d) 


User 

Response  i 

9 

Ho. - - - 

10 

Would  be  good  if  had  Center  boundaries  then  could  use  for  more  of  the  airspace. 
Go  farther  back  in  time  with  animation;  would  like  to  see  more  history  and  less 
projection  (only  need  out  to  30  minutes).  Would  like  to  see  one  display;  could 
replace  ITWS. 

11 

Would  like  to  see  incorporated  into  ITWS,  but  can  get  too  many  windows 

12 

Have  county  overlays. 

9.  Do  you  have  any  other  comments  or  concerns  regarding  the  forecast  product? 


User 

Response 

1 

Outstanding  system  that  needs  to  be  implemented  at  other  locations. 

2 

Ha  “  - - - 

3 

When  TCWF  is  used,  it  is  effective  and  trustworthy. 

4 

Had  questions  about  the  extent  of  weather  in  the  200  nautical  mile  range.  Does 
weather  go  out  that  far? 

5 

Really  like  it. 

6 

Has  performed  as  advertised. 

7 

NA  “  - - 

8 

Federal  Express  is  a  big  customer;  TCWF  has  been  beneficial  for  late  evening 
arrival  pushes.  Would  be  beneficial  for  Federal  Express  to  have. 

9 

Ha  ~  - - - 

10 

Lines  on  maps  are  too  thin;  improve  the  maps. 

11 

~NA  — - 

12 

Anything  is  an  improvement,  but  overall  has  not  used  much. 

f 


f 
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APPENDIX  C 


SAMPLE  MODIFIED  COOPER  HARPER  (MCH)  WORKLOAD  SCALE  MEM  TRACON 

Operator  mental  workload  is  one  component  used  to  assess  the  suitability  and  effectiveness  of 
an  operational  system.  In  order  to  ensure  that  mental  workload  for  the  Air  T raffic  Control  Traffic 
Management  Coordinator  (TMC)  or  ATC  Supervisor  has  not  increased  with  use  of  other 
systems  product,  the  Modified  Cooper  Harper  (MCH)  workload  scale  is  being  administered  to 
measure  overall  operator  mental  workload.  The  MCH  scale  will  be  administered  twice:  1)  prior 
to  Demonstration  conduct,  and  2)  post-  Demonstration  conduct. 

■  •  7,  Instructions;.  -  ml 


Consider  your  perceived  workload  given  moderate  to  severe  convective  weather  conditions. 
After  reading  the  difficulty  level  and  operator  demand  level  descriptions,  please  indicate 
workload  ratings  for  each  of  the  ATC  tasks  by  circling  the  number  (1-10)  on  the  right  hand  side 
of  the  scale.  Based  on  these  descriptions,  choose  the  rating  number  that  most  closely  matches 
your  perception  of  mental  effort  in  performing  these  tasks.  Each  task  is  listed  on  a  separate 
page  accompanied  by  the  workload  rating  scale.  Please  read  the  descriptions  on  difficulty  level 
and  operator  demand  level  carefully  before  circling  your  rating  score. 
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1.  Change  runway  configuration 

DIFFICULTY  LEVEL  OPERATOR  DEMAND  LEVEL  RATING 


VERY  EASY,  HIGHLY 
DESIRABLE 

OPERATOR  MENTAL  EFFORT  IS  MINIMAL  AND  DESIRED 
PERFORMANCE  IS  EASILY  ATTAINABLE 

1 

EASY, 

DESIRABLE 

OPERATOR  MENTAL  EFFORT  IS  LOW  AND  DESIRED 
PERFORMANCE  IS  ATTAINABLE. 

2 

FAIR, 

MILD  DIFFICULTY 

ACCEPTABLE  OPERATOR  MENTAL  EFFORT  IS 

REQUIRED  TO  ATTAIN  ADEQUATE  SYSTEM 
PERFORMANCE 

3 

MINOR  BUT 

ANNOYING 

DIFFICULTY 

MODERATELY  HIGH  OPERATOR  MENTAL  EFFORT  IS 
REQUIRED  TO  ATTAIN  ADEQUATE  SYSTEM 
PERFORMANCE 

4 

MODERATELY 

OBJECTIONABLE 

DIFFICULTY 

HIGH  OPERATOR  MENTAL  EFFORT  IS  REQUIRED  TO 
ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE. 

5 

VERY 

OBJECTIONABLE  BUT 

TOLERABLE 

DIFFICULTY 

MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE. 

6 

MAJOR  DIFFICULTY 

MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  BRING  ERRORS  TO  MODERATE  LEVEL. 

7 

MAJOR  DIFFICULTY 

MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  AVOID  LARGE  OR  NUMEROUS  ERRORS. 

8 

MAJOR  DIFFICULTY 

INTENSE  OPERATOR  MENTAL  EFFORT  IS  REQUIRED  TO 
ACCOMPLISH  TASK,  BUT  FREQUENT  OR  NUMEROUS 
ERRORS  PERSIST. 

9 

IMPOSSIBLE 

INSTRUCTED  TASK  CANNOT  BE  ACCOMPLISHED 
RELIABLY. 

10 
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